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PURPOSE AND RATIONALE 


In 2020, the Foundation for Child Development released Getting it Right: Using Implementation Research to 


Improve Outcomes in Early Care and Education. The publication provides insights into the value of including 
implementation research in the study of early care and education (ECE) and its potential to improve programs and 
policies and achieve stronger outcomes for all young children. As more programs are brought to scale, our ability to 
achieve greater impacts for all young children rests on a more nuanced understanding of the context in which ECE 
programs and policies are being implemented locally and the different impacts on specific subgroups of children. As 
a result, now is the moment for ECE researchers, practitioners, and policymakers to identify the critical components 
of effective ECE programs and policies and engage in meaningful exploration of what works, for whom, and under 


what conditions. 


This conversation guide was created to help faculty in institutions of higher education use Getting it Right as a 
resource in the preparation of future researchers to conduct implementation research. This guide supports higher 
education faculty to initiate and sustain conversations with students about effective early care and education, what 
still needs to be learned, and how implementation research can help fill the gaps. This guide focuses on faculty 


who are teaching a variety of courses, such as research design and methods, program evaluation, and statistics. 


As you and your students consider research questions, study designs, the interpretation and implications of findings, 
and the application of research to practice, you will be joining the authors of the original volume in the challenge 

of gaining a deeper understanding of the context in which research is conducted. To enhance your conversations, 
we interviewed each author and have included quotes that capture their insights about their work and the role of 
implementation research in the field. As these quotes reveal, the authors respect your students as future collaborators 


in moving the field forward. 


Many of the authors offer insights not only about implementation research design and approach, but also about 
the stance of implementation research based on their experiences. Stance refers to the attitudes, values, and beliefs 
each researcher brings to their work, as well as the skills and mindset required to be a continuous learner and/or 
collaborator in research. It is important to recognize that “stance” can also be influenced by unconscious bias. With 
this guide, you can invite each of your students to be intentional in exploring and establishing their stance. Such 
reflections will help ensure that long after your class, students will be aware of how who they are and what they 


bring to their research can impact their work. 
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USING THIS GUIDE AND HOW IT IS ORGANIZED 


This guide is organized for ease of navigation as you plan and teach your classes. It is divided into three sections 


that are intended to provide an overview of an implementation research approach; offer questions to support you 
in sparking and sustaining conversations with your students, both those studying research evaluation and methods 
and those whose focus is on ECE program quality and child development; and present reflections on moving ECE 


implementation research forward. 


Each chapter includes: 
¢ A summary excerpted from the Getting it Right Chapter Summaries 
* Themes excerpted from the Getting it Right full publication 


* Quotes from the author(s) 
* Questions for further discussion 


¢ A “Moving forward” section to facilitate further insights and takeaways 


The chapters open with a summary paragraph, which appear within the Getting it Right: Using Implementation 
Research to Improve Outcomes in Early Care and Education Chapter Summaries resource. The excerpts you will see 


throughout the chapters are from the full publication, Getting it Right: Using Implementation Research to Improve 
Outcomes in Early Care and Education. They are indicated by the corresponding page number. Discussion questions 
follow each excerpted theme and chapters close with the “Moving forward” insights. 


Discussions may lead to new questions or reflective exercises and activities that can deepen students’ understanding 
of implementation research and their role as researchers. You may decide to share this guide with your students as 
a resource to reflect upon during and after taking your class. By doing so, you help ensure the conversation 
continues as students have access to both content and reflections from each author about the rich possibilities of this 


research approach. 


Resources are also provided to help further engage your students with the publication content. In the full publication, 
Chapter 7 is a case study that highlights the experience of Boston Public Schools in building and refining its 
Preschool-2nd grade program. In this guide, Chapter 7 was used to generate a series of activities that you may 
choose to use with your students individually or in small groups. Yet, any chapter and accompanying discussion 


questions can be used to create group activities or other follow-up assignments. 


A complimentary Getting it Right 2020 Summer Webinar Series is available. You may use it to support your students in 


understanding the importance of implementation research, its role in the effectiveness of ECE interventions, and related 
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themes from the companion publication. Students may enjoy seeing the publication authors reflect on their work and 


hear the discussions. Please access these highlights and key takeaways on-demand as you explore this guide. 


Getting it Right: Using Implementation Research to Improve Outcomes in Early Care and Education 
Virtual Launch 


In this webinar, key authors shared an exclusive overview about the publication, chapter highlights, insights about 
the value of implementation research in ECE, and its potential to improve programs and policies for stronger 


outcomes for all young children. 


Featured Presenters: 

Margaret R. Burchinal, Ph.D., University of North Carolina at Chapel Hill 
Jeanne Brooks-Gunn, Ph.D., Columbia University 

Tamara G. Halle, Ph.D., Child Trends 

JoAnn Hsueh, Ph.D., MDRC 

lheoma U. Iruka, Ph.D., HighScope Educational Research Foundation 


Jacqueline Jones, Ph.D., Foundation for Child Development 


Discussant: Jason Sachs, Ed.D., Boston Public Schools 


Getting it Right: Part 1: What more do we need to know about high-quality ECE programs 


In this webinar, presenters highlighted key takeaways from the publication as it attempts to answer several questions 
for the field: What instructional content and strategies are tied to positive child outcomes? What more do we 

need to know about supporting dual language learners and bilingual education? What elements of coaching 

lead to changing teacher practice? How can we build effective and sustainable systems of ongoing professional 
development? How can quality programs be brought to scale? How can we ensure that programs are scaled in a 
way that promotes the development of children from diverse racial/ethnic, socioeconomic, cultural, and linguistic 


backgrounds? 


Featured Presenters: 

Linda Espinosa, Ph.D., University of Missouri-Columbia 
Dale Farran, Ph.D., Vanderbilt University 

Jacqueline Jones, Ph.D., Foundation for Child Development 
Robert Pianta, Ph.D., University of Virginia 

Jason Sachs, Ed.D., Boston Public Schools 


Discussant: Ellen Frede, Ph.D., National Institute for Early Education Research 
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Getting it Right: Part 2: Implementation Research in Early Care and Education 
In this webinar, presenters highlighted key takeaways from the publication as it attempts to answer several questions 


for the ECE field: What can we learn from implementation research principles to lead ECE programs, practices, and 
policies to better outcomes for young children? How can various implementation research designs address questions 
relevant to the field? How is improvement science different from implementation science? How are qualitative studies 
helping us understand variation across sites and localities implementing evidence-based programs? How can equity- 


focused implementation research be an effective tool for reducing bias in evaluations? 


Featured Presenters: 

Tamara Halle, Ph.D., Child Trends 

JoAnn Hsueh, Ph.D., MDRC 

Milagros Nores, Ph.D., National Institute for Early Education Research 
Sharon Ryan, Ed.D., Rutgers University 

Sara Vecchiotti, Ph.D., Esq., Foundation for Child Development 


Discussant: Caroline Ebanks, Ph.D., Institute of Education Sciences 


Getting it Right: Part 3: Moving Towards Equity Through Implementation Research 
In this webinar, presenters shared insights about how equity-focused ECE research attempts to address the needs of 


children from diverse racial/ethnic, socioeconomic, cultural, and linguistic backgrounds. The following questions are 
addressed: What strategies does an equity-focused implementation research study employ across various stages of 
research? How can a Social Determinants of Early Learning framework be used to guide researchers in addressing 
systemic inequities? How can ECE programs foster early bilingualism while improving outcomes for children? What 


can we learn from Boston Public Schools’ experience in moving towards achieving equitable outcomes? 


Featured Presenters: 

Linda Espinosa, Ph.D., University of Missouri-Columbia 

lheoma U. Iruka, Ph.D., HighScope Educational Research Foundation 
Jacqueline Jones, Ph.D., Foundation for Child Development 

Milagros Nores, Ph.D., National Institute for Early Education Research 
Jason Sachs, Ed.D., Boston Public Schools 


Discussant: Kristine Andrews, Ph.D., Child Trends 
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In closing, while all these resources are provided to support student learning, you know your students. You know 


their interests, strengths, and the areas they need to further develop. This guide is written for you. Make it yours. 


WHAT IS IMPLEMENTATION RESEARCH? 


Getting it Right: Using Implementation Research to Improve Outcomes in Early Care and Education does not prescribe 


a single definition of implementation research. Instead, it draws attention to the rich potential of investigating ECE 
programs in context. It also outlines how implementation research can advance the ECE field by answering questions 
that policymakers and practitioners prioritize as they seek to continuously improve or strengthen the ECE policies and 


programs that they govern, manage, and implement to ensure positive outcomes for all young children. 


In creating its publication, the Foundation for Child Development brought together a group of respected, thoughtful 
researchers to share their perspectives, questions, and experiences with a rigorous scientific approach that presents 
both opportunities and challenges. Many of them have conducted groundbreaking research that now serves as 

the foundation of our knowledge of what it takes to offer high-quality experiences to young children. As global 
attention has focused on issues of racial and social justice, we see from this foundational research that issues of 
equity and access to food, health care, housing, and high-quality early experiences have been a part of the early 
childhood landscape for a very long time. Implementation research provides an opportunity to build on our research 
foundations; explore root causes; examine our research methodology; and build stronger connections across 


research, policy, and practice. 


As Getting it Right demonstrates, conducting sound, rigorous, high-quality ECE implementation research to build 
evidence for the field is no easy task. Realistically, researchers doing such work need to be willing to “embrace 
the messy,” from initial design through final analysis, interpretation, and dissemination. The messiness reflects the 
intricacies of the interventions and is precisely what makes the work so interesting (Vecchiotti, Conclusion). It helps if 


researchers assume a stance in which they: 


¢ Have a deep appreciation for ever-evolving contexts, typically encompassing multiple layers of policy 


and programmatic decisions and surrounding conditions. 


¢ Have extensive knowledge about various rigorous designs and methods of analysis to answer 


nuanced and interrelated questions nested within and across these implementation contexts. 
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* Consider building a collaborative process with policymakers and practitioners that encompasses the 
entire research process, including the co-construction of research questions, and continues throughout 
research design, data access, collection, and reporting. This is an important shift for applied 
researchers, and the benefits of this two-way relationship are many. Furthermore, interpreting the data 
and determining the implications can help guide collaborative thinking about how to account for 


particular implementation contexts and provide more insights into research-to-practice connections. 


Consider developing their own knowledge about their research partner's work, especially because 
they are examining the tensions between the planned ideal and actual implementation. To make 
meaningful and useful policy recommendations, researchers need to acquire operational knowledge, 


such as understanding a program’s or policy's specific purpose, elements, and process. 


Provide a common ground of shared operational knowledge to help build and maintain trust among 
collaborators throughout all stages of research. It can also help to manage appropriate expectations 


regarding what research can do to influence or support continuous quality improvement efforts. 


With such knowledge, skills, and dispositions, applied implementation researchers can increase the potential of 
research to shape, improve, or transform ECE policy and programs in ways that allow these programs to better 


serve all children and their families. 


FILLING IN THE GAPS OF OUR KNOWLEDGE TO IMPROVE OUTCOMES FOR ALL CHILDREN 


Researchers have largely considered the randomized control trial (RCT) to be the gold standard of experimental 


design in understanding and assessing program effectiveness. This design randomly assigns some children to a 
group that receives a defined treatment and others to a group that does not. Assuming that the two groups and 
all other things are equal, posHreatment differences between the two groups are assumed to be caused by the 
treatment’s impact. Yet these trials may not capture nuances of variation or answer critical questions, including: 
Are all children experiencing the program under the same conditions? Are specific subgroups of children 


demonstrating different responses to the intervention? 


Additional robust quantitative and qualitative data are also needed to ensure stronger outcomes for all children 

and significantly narrow the opportunity and achievement gap for minoritized children and those living in poverty. 
A deeper understanding is needed about how equity issues, especially racism and poverty, discrimination related to 
race/ethnicity, socioeconomic status, cultural and linguistic background, gender, and immigration status, influence 
implementation of early childhood programs and outcomes for children. It may be that these issues are central to 
understanding the relationship between populations most in need of services and components or constellations 

of program components. Understanding this is at the foundation of informed decisions about how to design and 


implement programs on behalf of all children. 
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Getting it Right does not propose implementation research as a replacement for the RCT, but it can augment RCTs. 

In fact, implementation research designs often use both quantitative and qualitative data sources to fully describe the 
unfolding stages of implementation and changing contexts (Halle, Ch. 10; Hsueh & Maier, Ch. 9; Ryan, Ch. 11). To 
fully capture how ECE programs and policies influence young children’s development, we must pay attention to both 


outcome- and implementation-oriented research. 


HOW IMPLEMENTATION RESEARCH CAN SERVE THE ECE FIELD 


Five key contributions that implementation research can make to the ECE field include: 


Implementation research augments RCTs as it goes beyond answering the question of whether effects are 
demonstrated to explaining why or why not. By doing so, implementation research illuminates what makes ECE 
programs, practices, and policies effective and why. In this way, it can help support policies and practices around 
program replication, expansion, and sustainability, and guide program improvement to ensure that ECE programs 


reach their potential for narrowing achievement gaps. 


Implementation research can push ECE research forward by identifying deeper questions about the 
multifaceted root causes of inequity and ways to eliminate disparities. By understanding what's working, 
what isn’t, and why, with the aim of advancing equity across children and families, research can strongly support 

the development of effective programs and policies for all children. To be effective and to minimize the introduction 
of bias into the research itself, research integrates equity concepts across all research components, including 
questions asked and interpretation and dissemination of findings. Effective implementation research incorporating 
equity concepts depends in large part on recognizing the experiences, values, and mindsets of researchers and 
collaborators as individuals. The field urgently needs an implementation research approach to guide localities on the 
specific challenges and opportunities they may encounter as ECE programs are implemented in diverse real-world 


settings (Weiland, 2018’). 


Implementation research encourages refining the measurement of ECE quality to help achieve 

higher program quality and better outcomes for children. Typically, the field focuses on global measures 

of ECE quality, despite modest associations with child outcomes (Burchinal & Farran, Ch. 1). This includes process 
quality (e.g., the interactions between educators and children) and structural quality (e.g., caregivers’ education 
and training, wages and benefits, the ratio of children to caregivers, the number of children in a setting, program 
leadership and administration, and parental involvement’). Focusing instead on measures and program models 
that concentrate on specific instructional content and strategies to promote children’s school readiness skills related 
to language, executive functioning, and self-regulation would be a step forward. To this end, tools will be needed 
to take a fine-grain look at what happens between teachers and specific subgroups of children and what works 


for children. 
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Implementation research potentially can provide timely answers to policymakers’ questions as it 
enhances and extends findings that are useful and meaningful for continuous quality improvement 

of systems and practices (Halle, Ch. 10). Implementation research should be rigorous and complex, while 
also taking a practical approach. It examines program implementation in real time, supporting policymakers and 
the field to consider contexts and other variables that influence quality and outcomes for subgroups of children. 
An intriguing tool, implementation research has the potential to provide needed information if policymakers and 


the early childhood community are going to make the best decisions for all children. 


Implementation research provides data needed to address implementation scale-up questions, 
particularly how and when ECE is effective and for whom. ECE programs and policies are increasingly being 
brought to scale, particularly in states and municipalities (Friedman-Krauss et al., 2019°). Yet as research shows, 
many evidence-based ECE models have proven to be insufficient to guide program scaling that successfully benefits 
all children. Implementation research is a systematic inquiry into how a program is received and experienced in 
real-world settings and situations. In the long term, if we cannot answer implementation scale-up questions related to 
how and when ECE is effective, we risk losing support for increased investment in ECE (Jones & Vecchiotti, 2020), 
because expectations for ECE to attain certain child outcomes might outstrip results (Brooks-Gunn & Lazzeroni, 


Ch. 2). Implementation research can help minimize this risk. 


IMPLEMENTATION RESEARCH AND YOUR STUDENTS: MAKING THE LINK 


Your students may already be conducting or using research, or they might do so in the future. Either way, they have 


an opportunity to contribute to deepening the understanding of implementation research as they raise questions and 


share experiences during course discussions. 


Those who are writing dissertations may consider using implementation research as they design their study and 
interpret and apply their findings. The insight that ECE research involves having rigorous conversations about 


questions that still need to be answered is important for students and the field to acknolwedge. 
The future of your students—whether they are studying program evaluation or research methods—and the ECE field 


are filled with research possibilities. In the words of Pianta (Ch. 5): “Implementation science can provide the young 


researcher with an intellectual home.” (Interview, 11/9/2020) 
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Questions continue to emerge, suggesting an urgent need to expand applied research to gain a more nuanced 


understanding of how programs and policies are implemented and influence subgroups of children. Answering these 


questions will require a genuine collaboration between researchers and practitioners in all roles, and often families 


and community members as well. 


The goal of this guide is to support you to integrate implementation research into your curriculum and initiate 
conversations with your students. The guide can help to encourage all students to reflect upon these emerging 


questions about implementation research. 
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SECTION 1 


WHAT DOES RESEARCH TELL 
US ABOUT EFFECTIVENESS 


AND IMPLEMENTATION OF ECE 
PROGRAMS ACROSS THE 
BIRTH-TO-EIGHT CONTINUUM? 


IN SECTION 1: 


Chapter 1: What Does Research Tell Us About ECE Programs? 
By Margaret R. Burchinal, Ph.D., University of North Carolina at Chapel Hill and Dale C. Farran, Ph.D., Vanderbilt University 


Chapter 2: What Are Reasonable Expectations for ECE Program Effectiveness? 
By Jeanne Brooks-Gunn, Ph.D., Teachers College and College of Physicians and Surgeons, Columbia University and Sarah Lazzeroni, Teachers College, Columbia University 


Chapter 3: Using a Social Determinants of Early Learning Framework to Eliminate Educational Disparities and Opportunity Gaps. 
By Iheoma U. Iruka, Ph.D., HighScope Educational Research Foundation 


16 GETTING IT RIGHT: THE CONVERSATION GUIDE FOR PREPARING THE NEXT GENERATION OF IMPLEMENTATION RESEARCHERS FOUNDATION FOR CHILD DEVELOPMENT 


SECTION 1, CHAPTER 1 


WHAT DOES RESEARCH TELL US 
ABOUT ECE PROGRAMS? 


Margaret R. Burchinal, Ph.D., University of North Carolina at Chapel Hill 
Dale C. Farran, Ph.D., Vanderbilt University 
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In What Does Research Tell Us About ECE Programs?, Margaret Burchinal and Dale C. Farran 
summarize the extensive research relating early care and education (ECE) quality to children’s 
short- and long-term development. In discussing the factors that limit current ECE programs 

and policies from promoting better outcomes, they find that the field often focuses on current 
measures of global ECE quality despite very modest associations with child outcomes. Their 
interpretation of the research suggests focusing on program models that concentrate on 
specific instructional content and strategies to promote children’s school readiness skills related 
to language, executive functioning, and self-regulation. Such a program approach is likely to 
be more successful in supporting the long-term development of all children. 


The summary above appears in the Getting it Right Chapter Summaries resource. 


We see these very modest relationships between indicators of quality and outcomes, yet we haven’t 
wanted to question underlying assumptions of the ECE field about how they are connected. We need 
to take an evaluation point of view and ask: What is the goal and how is meeting this goal being 
implemented? What did you do to meet that goal? Are you doing what you think you are doing? 
What is the outcome on children? (Burchinal interview, 10/9/2020) 


The following themes, chapter excerpts from the Getting it Right full publication, and related questions are provided 


to stimulate additional engagement around the ideas and inquiry shared in this chapter. 


> 1. Structural Quality ™ Process Quality ™ Children’s Outcomes. Most ECE research is based on a 
theoretical model that posits that structural quality (e.g., characteristics such as teacher education and ratio of children 
to adults) lays the foundation for process quality (i.e., the frequency and quality of interactions between caregivers and 
children), and that it is process quality that impacts child outcomes. But the evidence supporting this model using current 
measures of structural and process quality is quite limited. Thus, we do not know enough about what works (or not), for 


whom, and under what conditions in promoting which skills for young children (p. 16). 
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Current ECE quality models assume that children acquire cognitive, academic, and social skills when they experience 
high levels of process quality, but the models do not specify how quality experiences promote specific skills. The fact 
that we see much larger impacts on outcomes in studies of specific curricula (Duncan & Magnuson, 2013) than in 


studies of ECE quality (Burchinal, 2017) suggests that ECE can produce substantial gains in specific outcomes when it 
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promotes those outcomes with evidence-based practices (p. 27). 


Most of our quality measures focus on what a teacher is doing. Thinking about the child might give 
us a better picture. We want to measure how children in a setting may have different experiences. 
For example, a teacher may be focusing on a few kids—those who engage with her... The teacher 

may be doing wonderful things, but not with all children. In that case, it is important to have your 


quality measure reflect the facts that only some children are benefiting from good teaching practices. 


(Burchinal interview, 10/9/20) 


Discussion Questions 


Research Design and Methods 


iF 


How should researchers understand and move forward with a paradigm shift in 


understanding relationships between quality definitions and child outcomes? 


. How should researchers understand and move forward with a paradigm shift, that is, 


moving from a bird’s eye view of quality and child learning to an on-the-ground view of 


seeing how specific curricula and activities foster specific outcomes? 


. How does this paradigm shift influence how quality and child outcomes are measured? 


¢ What types of adaptations are needed and what types of new 


measures need to be created? 


. What are some examples of implementation research questions surrounding structural 


and/or process quality that can help us better understand quality, instructional content 


and strategies, child outcomes, and the relationship between them? 


* Do we expect the answers to these questions to vary by setting, type of 


service, or age of children? 


¢ Why and how are these differences important in implementation 


research specifically? 
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> 2. Specific aspects of ECE quality appear to enhance children’s early development. Preschoolers 
showed modest but statistically significant gains in academic and social skills when they experienced more frequent, 
warm, and responsive interactions with caregivers (Mashburn et al., 2008; NICHD ECCRN, 2002; Raver et al., 
2011). Gains in academic skills are modestly larger when instruction includes detailed feedback, and sequenced 
and elaborated support for learning (Howes et al., 2008; Mashburn et al., 2008). Language and academic skills 
were higher when caregivers encouraged children to talk and engaged in multi-turn conversations that elaborated 
on a given topic (Justice, Mashburn, Pence, & Wiggins, 2008; Wasik & Hindman, 2011). Finally, gains in language 
and social skills were larger when children were offered a wide range of age-appropriate activities such as reading 
with adults, pretend play with peers, and play with books, blocks, water, and sand, demonstrating gains in language 
and social skills (Sylva et al., 2012) (p. 19-20). 


Don’t start with an assumption of what is quality. Be strategic. We need to think carefully about 
what skills we want ECE to promote, identify which activities promote those skills, and how can we 
implement them into programs in ways that engage children. (Burchinal interview, 10/9/2020) 


Discussion Questions 


Research Design and Methods 


1. What research designs can be used to investigate relationships between specific 


teaching practices focused on instructional content and strategies tied to child outcomes? 


* How are they different from a focus on structural or process quality? 


Research Collaboration and Partnerships 


2. Do different understandings of quality and child outcomes change the collaboration 
between researchers and stakeholders (e.g., administrators, leaders, teachers, 


caregivers, parents, etc.)? 


¢ Does it change the role or participation of partner stakeholders? 


Program Evaluation 


3. What approach should researchers take to understand a program’s purpose and design, 
as well as expectations of what early educators should know and be able to do to support 
children’s learning? Provide some examples of research questions examining how program 


administrators and leaders support early educators’ instructional practice. 
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> 3. Quality measures may need to focus more on the frequency and quality of 

intentional teaching. Recently, several measures have shown promise for expanding the measurement of ECE 
quality. They involve behavioral counts rather than ratings, and they vary in terms of whether the unit of observation 
is the teacher or multiple children in the classroom. Connor et al., (2011) developed an integrated system involving 
child monitoring, classroom observations, and instruction that has been shown to substantially improve reading skills 


in early elementary school; a preschool version is in the works. 


Observational measures that describe how children spend their time and how teachers interact with them appear 
promising. One, the Snapshot (Ritchie, Weiser, Kraft-Sayre, & Howes, 2001), describes how much time individual 


children spend in different types of activities in terms of content area and instructional format (p. 23). 


We are so strongly convinced that we know the answer, we don’t stop to question. Of course, if you 
can provide high-quality interactions, children are going to do better. But when you look at quality 
measures, we don’t see this. We’ve assumed that if you measure quality, that is what matters, but we 
don’t focus on outcomes and what leads to these outcomes. Do not assume the current paradigm and 
widely used quality measures are sufficient to improve child outcomes long term. Don’t be afraid to 
say we have improved quality but not outcomes. (Burchinal interview, 10/9/2020) 


Discussion Questions 


Research Design and Methods 
1. What underlying constructs do these measures lend themselves to? 
¢ What are the measurement gaps? 


2. How do available measures align with the various understandings of quality (e.g., 


structural, process, and specific instruction content and practice)? 


3. How can researchers combine or alter available measures to best evaluate quality in a 
way that relates to positive, short- and long-term child outcomes? Provide some examples of 


instructional content, strategies, and supports that can be used to measure children’s learning. 


4. How does the interpretation of the results of quality measurements vary based on setting, 


age, race, gender, etc.? 


¢ How does that interpretation influence the study of implementation? 
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5. What still needs to be understood in measures of how children and early educators 


interact to enhance our understanding of quality and child outcomes? 


Program Evaluation 


6. What are the roles of members of the ECE workforce (e.g., teachers, assistant teachers, 


classroom aides, directors) in data collection for measures? 


¢ How much administrative demand is placed on the workforce as part 


of quality assessments, observations, and measures? 


* How could researchers help reduce this demand, either in measure 


design or data collection practices? 


7. What types of studies could be designed to improve supports available to teachers to 
improve instructional practice? 


8. What types of studies could be designed to improve the way preschool programs 


implement instructional practices to meet desired child outcomes? 


9. How can researchers work with stakeholders to improve teacher salary, turnover, 
and retention? 


> Moving forward. 


“If we want to improve the lives of children, we need to look with fresh eyes at the data and 


understand what we are doing well and need to do differently.” (Burchinal interview, 10/9/2020) 


Discussion Questions 


Research Approach and Stance 
1. What does it mean to look at the data with “fresh eyes” 2 


2. What knowledge, experiences, and assumptions about early care and education might 
you have to set aside? 


3. How might looking at data with fresh eyes influence your work as a researcher? 
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FOR ECE PROGRAM EFFECTIVENESS? 


Jeanne Brooks-Gunn, Ph.D., Teachers College and College of Physicians and Surgeons, 
Columbia University 
Sarah Lazzeroni, Teachers College, Columbia University 
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In What Are Reasonable Expectations for ECE Program Effectiveness?, Jeanne Brooks-Gunn and 
Sarah Lazzeroni set a framework for reasonable expectations of early childhood education 
(ECE) program effectiveness given the great variability in quality, resources, duration, and 
children served. While comprehensive, high-quality ECE programs hold the promise of large 
effects for children at risk and very high returns on investment, the authors put forward what 
can be reasonably expected from programs under present conditions as policymakers and 
practitioners manage systemic changes to achieve ideal quality and outcomes. 


The summary above appears in the Getting it Right Chapter Summaries resource. 


Our chapter focuses on looking beyond whether an effect is significant or not to focus on a pattern of 
findings and range of effect sizes by program, outcome, treatment dose, and comparison group. When 
we see patterns and are realistic, we can make a difference—we can provide one piece of the puzzle. 
(Brooks-Gunn interview, 10/9/2020) 


The following themes, chapter excerpts from the Getting it Right full publication, and related questions are provided 


to stimulate additional engagement around the ideas and inquiry shared in this chapter. 


> 1. What are reasonable expectations for ECE program effectiveness? The overlap is evident in that 
asking about expectations raises questions about what is reasonable today given the state of ECE quality, as well as the 


variability in quality. In general, ECE program impacts are expected to be small-to-medium, but not large! (p. 39-41). 


Today, children from low-income families have access not only to Head Start but also, in many cities and states, to 
universal pre-K programs, often run by or in collaboration with a department of education. Other partially subsidized 
programs also exist (some funded through the Child Care Development Block Grant). The two best-known, small-scale 
evaluations, the Perry Preschool and Abecedarian projects, also were initiated in the 1960s and 70s and also targeted 
poor children; very few of the children in the control groups received any other preschool experiences (Belfield, Nores, 
Barnett, & Schweinhart, 2006; Campbell, Ramey, Pungello, Sparling, & MillerJohnson, 2002; Heckman, Moon, Pinto, 
Savelyev, & Yavitz, 2010; Hill, Gormley, & Adelstein, 2015). All of this suggests that the counterfactual for treatment 
today is different from what it was previously (p. 45). 
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If you don’t know the context, a program’s theory of change, all the things that naturally go 
wrong when you implement any program, your expectations are likely not realistic... When you have 
a program that shows a modest effect size and you expected a moderate-to-large effect size, you 
may think it failed. That is not true. You do make a difference. Don’t be discouraged. Instead, go on! 
(Brooks-Gunn interview, 10/9/2020) 


Consequently, it may be unreasonable to expect effect sizes today that are similar to those in the past if most children 
are now receiving some ECE at three and four years of age. This does not mean that preschool is ineffective. It just 
means that traditional evaluations of treatment and control will find smaller effect sizes, since most children in the control 


group are receiving some sort of preschool (p. 46). 


Discussion Questions 


Research Design and Methods 


1. How does the history of the belief in the efficacy of early education intervention for 


children living in poverty shape ECE research today compared to the past? 


2. How are the small, early-model ECE programs different than the ECE programs of today 


(e.g., children served, services provided, program staffing)? 


* How would your answers to these questions influence research 


questions and designs? 


3. What do we understand about the short- and long-term effectiveness of the small, early- 


model programs vs. contemporary programs? 
* How have the research questions changed over time? 
4. How has ECE availability as a context for research changed over the last 50 years? 
¢ Are there implications for expected outcomes for children and 
conceptualizations about ECE effectiveness? 
5. Considering community context for research design and implementation, do different 


stakeholders emphasize similar or unique factors? 


¢ What are examples of factors that stakeholders may emphasize over others? 
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Program Evaluation 


6. What child outcomes should be reasonably expected in the short- and long-term due to 
increased ECE participation as a result of the variety of preschool options? Explain how these 
expectations might change within the context of a program that you are familiar with (either local 


or not). 


¢ What factors will influence the likelihood that expectations for child 


outcomes are met? 


7. How does understanding community context (e.g., location, program history and design, types of 


children served, resources) influence expectations of program effectiveness and research design? 


* How does community context change the expected impact on child 


outcomes and the way research results are interpreted? 


8. What does the apparent change in counterfactuals mean for how researchers determine 


counterfactuals for comparisons? 


¢ What does this change mean for the way researchers assess effect sizes for 


impact and explain outcomes to stakeholders? 


¢ Are there different counterfactuals that should be considered when asking 


different research questions? 


> 2. Describing and examining effect sizes (p. 47-49). It is sometimes difficult for the public, policymakers, 
and educators to understand what an effect size means. For example, does an effect size of .40 on early indicators of 
achievement for low-income students mean they'll do better in elementary school, and how much better compared to 
high-income students? The same question, of course, could be asked for dual language learners or for minority students. 
Two approaches can help translate effect sizes into more concrete indicators. The first is to explain what might be 


seen in a classroom where low-income students’ performance was one standard deviation below that of high-income 
students (p. 47). 


Having a robust effect size is also important given the expected reduction in effect sizes throughout the elementary 
school years. Without additional services or improvements to early elementary school, the effect of ECE will fall to one- 
half of its initial size by the end of third or fourth grade. Therefore, an effect size of one-half will become one-quarter 
and an effect size of one-third will become one-sixth. Effect sizes that are lower than one-third are very unlikely to be 


sustained into the late elementary school years (p. 40). 
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Another approach is to take an effect size at the end of a preschool intervention and estimate the increase in the 
number of children graduating from high school or college or predict kindergarten achievement scores to high school 


achievement scores (p. 48). 


Don’t look at one effect size from one evaluation and think you have an answer. Look for patterns 
over time. Be realistic. Think context. (Brooks-Gunn interview, 10/9/2020) 


Discussion Questions 


Statistics | Program Evaluation 
1. How would you explain effect sizes to different stakeholders who are unfamiliar with the term? 


¢ Would this explanation change when speaking to policymakers, as 
opposed to administrators or teachers? Bring your explanation to life using 


an example from your work or studies. 
2. What is the statistical and practical significance of having a robust effect size? 


3. How do we interpret findings of small, medium, and large ECE effect sizes within the 


context of specific programs? 


4. What factors should be considered when interpreting different effect sizes across 


different programs? 


5. What are meaningful effect sizes in relation to meeting expectations of program 
effectiveness and child outcomes for a program in your local context or a program 


you are familiar with? 


Research Design and Methods 


1. What types of research studies could you design to ensure that these meaningful effect sizes are 


sustained as children move through the K-12 system? 


2. What factors might you want to consider, in terms of research design and implementation of the 


research results, when choosing developmental outcomes to measure? 
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> 3. Promoting success: A multilevel model. Almost all ECE evaluations have assessed 

individual children, typically those who received an intervention and those who did not via 
random assignment, waitlist, or eligible age for entrance into preschool. But some have 

used administrative data as well. One interesting approach is to analyze school- or district-wide data from 
standardized testing to look at differences in achievement levels. In this way, comparisons can be made across time 

to see whether an intervention implemented at the school or district level has increased mean scores or competency 
levels... Such a cohort comparison was used effectively in the county-level effort in Montgomery County, MD (Marietta, 
2010). The school district staff, after examining the proportion of the district’s high school seniors who were ready for 
college, set a goal of having 80% of a graduating class college-ready. Working backward, they defined their goals for 
classes of pre-K to third-grade children (p. 56-57). 


It will take a lot of time to come up with the answers we need and are looking for. Don’t be discouraged. 
There is no quick fix for anything that is as important as changing children’s lives for the better. 
(Brooks-Gunn interview, 10/9/2020) 


Discussion Questions 


Program Evaluation 


1. Looking at the list of Montgomery County reforms, what role did research play? 


¢ What research questions remain? 


¢ What factors contributed to the success of their implementation study 


approach? 
¢ How might various stakeholders have been included in the research? 


¢ Which reforms used in this project do you hypothesize had the 
biggest impact on the study results and why? 


2. What may be the benefits of coupling individual child data with schoollevel data? 


¢ What might prevent systemwide initiatives from taking this approach? 


¢ What could help ameliorate those factors? 
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> Moving forward. 


“We have framed our chapter in hopes that students will come away with a more realistic view of what 
we can expect. This is a new idea for many. Implementation research is about what really happens 
on the ground. It requires looking at what is really going on. And looking for patterns. It requires that 
researchers “get real.” (Brooks-Gunn interview, 10/9/2020) 


Discussion Questions 


Researcher Approach and Stance | Research Collaboration and Partnerships 


1. How might Brooks-Gunn’s suggestion to “get real” influence your work today and in the future? 


2. How might her suggestion change your view of what you can contribute, or as she says earlier, 


“your piece of the puzzle,” regarding the goal of ensuring strong outcomes for all children? 


3. What are the implications for reaching out to others and creating collaborations in terms of 


other researchers and stakeholders? 


4. What are the implications for collaborating with people across disciplines (e.g., economists, 


financial analysts, health care providers, etc.)? 
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EDUCATIONAL DISPARITIES 

AND OPPORTUNITY GAPS 


lheoma U. Iruka, Ph.D., Frank Porter Graham Child Development Institute 
at the University of North Carolina-Chapel Hill (formerly HighScope 
Educational Research Foundation) 
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In Using a Social Determinants of Early Learning Framework to Eliminate Educational Disparities 
and Opportunity Gaps, lheoma U. Iruka surfaces social policies and factors that maintain 
inequities and ensure early learning disparities. These structural factors limit resources and 
supports that directly impact children’s outcomes, especially for low-income and minoritized' 
children and their families. She argues that to truly address early learning inequities and 
disparities, we must recognize systems that invisibly maintain and perpetuate inequities (and 
conversely privilege) from housing to education. For early childhood education (ECE) programs 
to meet their goals, the field must engage in more thoughtful, meaningful, and racially 
responsive research focused on understanding the causes and solutions for learning disparities 
and gaps. This will require the ECE research community to take an equity perspective that 
includes diverse voices and perspectives, especially those from minoritized communities, to 
examine how social and structural determinants impact children’s outcomes. 


The summary above appears in the Getting it Right Chapter Summaries resource. 


The following themes, chapter excerpts from the Getting it Right full publication, and related questions are provided 


to stimulate additional engagement around the ideas and inquiry shared in this chapter. 


> 1. With this economic and social cost of underutilized human potential and capability, the 
achievement gap, which is a symptom of systemic discriminatory policies and laws, needs to be 
treated as a public-health crisis. In this chapter, we adapt a framework used by the public-health sector—Social 
Determinants of Health (SDoH)—to address inequities and support the well-being of U.S. citizens at a population level 
(e.g., infant mortality and morbidity, teen pregnancy, or smoking) to show how early learning can address the inequities 


in education (p. 65). 


For early childhood education to truly address early-learning disparities at the systems level, we propose adapting the 


SDoH framework to early learning, calling it Social Determinants of Early Learning (SDoEL) (see Figure 2). 


The Centers for Disease Control and Prevention defines social determinants of health as “the complex, integrated, 
and overlapping social structures and economic systems that are responsible for most health inequities. These social 
structures and economic systems include the social environment, physical environment, health services, and structural 
and societal factors. Social determinants of health are shaped by the distribution of money, power, and resources 


throughout local communities.” (p. 69) 


30 GETTING IT RIGHT: THE CONVERSATION GUIDE FOR PREPARING THE NEXT GENERATION OF IMPLEMENTATION RESEARCHERS FOUNDATION FOR CHILD DEVELOPMENT 


GETTING IT RIGHT GUIDE CHAPTER 3: USING A SOCIAL DETERMINANTS OF EARLY LEARNING FRAMEWORK TO ELIMINATE EDUCATIONAL DISPARITIES 
AND OPPORTUNITY GAPS 


Figure 2. Social Determinants of Early Learning. 
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As Figure 2 shows, the concept behind SDoEL is that socioeconomic and political contexts (e.g., social policies about 
housing and education) lead to individuals’ socioeconomic position (e.g., education, income, or occupation), which 


then impacts their resources and living conditions, greatly reducing some children’s opportunities to thrive (p. 70). 


Bringing a public health framework to ECE can change our lens and help us see how we can 
support all children. (Iruka interview, 9/10/20) 
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Discussion Questions 


Research Design and Methods 
1. Using your local context, create a graphic of a local SDoEL framework. Do the factors included 
mirror or differ from the one in Chapter 32 
¢ What factors are missing or need to be modified? 
2. How would you design and promote local research and programs to deliver on the promise of 


strong outcomes for children, especially children from minoritized communities? 


3. How can the SDoEL framework help ensure researchers ask questions that enable them to gain 
insights into the multiple layers of community and personal factors that influence a child’ life 


and school experience? 
¢ What supports might researchers need in order to ask the “right questions” 
in the “right way”? 
A. In what ways can implementation research be designed to ensure the whole systems approach 
in the SDoEL framework is taken into account? 


5. How can research inform the development of the SDoEL framework? 


6. Does the research that you are familiar with-from your studies or research you have 


participated intake this systems approach into account? How so? 


7. How can the SDoEL framework be used to identify racist and discriminatory policies 


and practices? 


8. How can it be used to address racist and discriminatory policies and practices? 


Program Evaluation 
1. Why is the idea of quality rating and improvement systems (QRISs) important in improving 
early learning programs? 
* How do QRISs take factors of the SDoEL into account? 


¢ What factors are they not taking into account? What role could research 


play in assessing QRISs and improving QRISs to benefit all children? 


2. What role could research play in understanding how QRISs can benefit all children? 
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> 2. ECE research must consider racism and discrimination using the SDoEL framework. For too 
long, most ECE research has indicated that many children of color and children from low-income households are not 
prepared for school and need early care and education programs. Unfortunately, most of the research, especially 
about children of color and their families, has been done with a deficit perspective, without consideration for the social 
determinants that lead to the disparities witnessed even after interventions. The results have often shamed and blamed 
children, families, and communities for low scores on language and cognitive assessments without considering the 


historical legacy of racism and discrimination and white supremacy that couches all aspects of early learning (p. 78). 


All of us bathe and swim in the water of racism. Yet most of us don’t see it or live it. If we don’t see 
something, we can’t measure it or learn about it. (Iruka interview, 9/10/20) 


Discussion Questions 


Research Approach and Stance | Research Design and Methods 


1. What does it mean to utilize a colorblind approach in research? 


¢ Why is this problematic? 
2. What roles do racism and discrimination play in the lives and early learning experiences of 


minoritized young children today and in their later outcomes? 


3. What role has racism and discrimination played in ECE research, not only theoretically but 
more concretely, as in the measures and assessments used, types of analysis conducted, ways 


data is interpreted and disseminated, etc.? 
4. What narrative changes are needed to advance racial, economic, and social equity in ECE? 


¢ What other types of changes are needed? 
5. How can various implementation research designs examine the root causes of racism and 
discrimination in policy and practice? 
* How can it help to identify ways to eliminate and mitigate racism and 
discrimination in ECE? 


6. How can researchers’ personal beliefs and experiences with racism and discrimination 


influence the questions that theorists use to frame a study? 
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7. What are ways researchers can mitigate potential bias and promote research grounded in 


equity principles? 


8. What approach can researchers take to engage in more cross-sector collaborative research to 


examine the SDoEL frameworke 


9. What research questions could build on the strengths and assets of marginalized children 
within racially, ethnically, socioeconomically, linguistically, and culturally diverse families? 
What research questions could help promote families’ and communities’ ability to thrive and 


promote all children’s learning? 


10. Using the SDoEL framework, what research questions could be used to analyze the way the 


ECE workforce is impacted by the same systems that lead to early learning disparities? 


Research Collaboration and Partnerships 


1. How can collaboration within research teams help mitigate racism and discrimination? 


> 3. Though access to early learning opportunities has increased, academic and social gaps 


by income and race/ethnicity have not been eliminated. Education scholars see some reduction in 


these gaps, but “at the rates that the gaps declined in the last 12 years, it will take another 60 to 110 years for them 


to be completely eliminated” (Reardon & Portilla, 2016, p. 12). Thus, early learning in isolation will not close the 


achievement gap in a timely way. Researchers, in partnership with practitioners and policymakers, must uncover and 


address the root causes of racial and economic disparities, and find research-based specific practices and policies 


that can eradicate these gaps and inequities (p. 67). 
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Discussion Questions 


Research Design and Methods 


1. What factors are leading to persistent achievement and opportunity gaps in young children’s 


learning and development? 


* How might such factors influence early educator-child interactions 
and instruction provided to children from various racial, ethnic, 


cultural, and linguistic backgrounds? 


* How might teachers respond to such factors in their instruction and 
interactions, depending on whether they have similar backgrounds as 


their students or not? 
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2. What questions could be added to address the relationships between achievement and 


opportunity gaps? 
* How do specific circumstances and characteristics of place, location, 
institutions, etc. influence relationships between the two gaps? 


* How do the interactions between the two gaps, and the context in 


which they sit, affect efforts to reduce them? 
3. How can various implementation research designs examine the multifaceted root causes 


leading to achievement and opportunity gaps? 


4. What does it mean to ensure equity, and not equality, when it comes to funding and resources 


for early learning programs? 


5. How can research be used to ensure equity and address disparities? 


Research Collaboration and Partnerships 


1. How can researchers, policymakers, and practitioners collaborate to advance equity 
throughout the research process, from co-construction of research questions to research design, 


data access, collection, interpretation, and reporting? 
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> Moving forward. 


[Is] it assumed that all children require the same amount and type of sensitive and cognitively enriching interactions and 


instruct 


their ra 


u“ 


Notes 


ions, without acknowledging their health, family and home condition, community environment, or narrative about 
ce or neighborhood (p. 81)? 


Humans change systems and systems change humans. Students are needed to be part of the change 
conversation. The questions they ask, their interactions with others are all opportunities to raise 


questions and to contribute to filling in the gaps of what we know.” (Iruka interview, 9/10/20) 


Discussion Questions 


Research Approach and Stance | Research Collaboration and Partnerships 


1. How can researchers, policymakers, and practitioners include children and their families in 


their work to raise questions, contribute knowledge, and inform systems change? 


2. How might collaborative thinking increase and enrich your insights into research-to-practice 


connections? 
3. What are the two most important points from Chapter 3 that resonate for you as a researcher? 


¢ What do you think makes these points meaningful for your work? 


* How will these insights influence your work? 


"Smith (2 


016) states that “groups that are different in race, religious creed, nation of origin, sexuality, and gender and as a result of social constructs have 


less power or representation compared to other members or groups in society should be considered minoritized.” People who are minoritized endure 
mistreatment and face prejudices that are forced upon them because of situations outside of their control. https://www.theodysseyonline.com/minority-vs- 
minoritize 
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SECTION 2 


WHAT STILL NEEDS 


TO BE UNDERSTOOD? 


IN SECTION 2: 


Chapter 4: Making Prekindergarten Classrooms Better Places for Children’s Development. 
By Dale C. Farran, Ph.D., Vanderbilt University 


Chapter 5: Improving Quality and Impact Through Workforce Development and Implementation Systems. 
By Robert C. Pianta, Ph.D., University of Virginia and Bridget K. Hamre, Ph.D., Teachstone 


Chapter 6: Addressing Equity in the ECE Classroom: Equal Access and High Quality for Dual Language Learners. 
By Linda M. Espinosa, Ph.D., University of Missouri-Columbia 


Chapter 7: Vignette: Building a High-Quality Program—the Boston Public Schools Experience. 
By Jason Sachs, Ed.D., Boston Public Schools 
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MAKING PREKINDERGARTEN 


CLASSROOMS BETTER PLACES FOR 
CHILDREN’S DEVELOPMENT 


Dale C. Farran, Ph.D., Vanderbilt University 
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In Making Prekindergarten Classrooms Better Places for Children’s Development, Dale 

C. Farran illuminates four prekindergarten classroom elements that lead to better child 
outcomes: listening to children, teacher/child interactions that encourage critical thinking, 
positive classroom environments, and children’s active engagement in learning. These aspects 
of classroom functioning often fall outside current quality ratings, curriculum assessments, 
and standards. Farran points to the need to recognize, analyze, and measure these critical 
interactions between children and teachers, as they can impact outcomes more than current 
standards and measures do. 


The summary above appears in the Getting it Right Chapter Summaries resource. 


If we are serious about investing more public money in pre-K, we need to come up with a composite 
set of measures for diverse teachers in diverse settings. As part of our composite set of measures, 
we need to pay attention to context of the setting. For example, in urban settings like New York City, 
pre-K programs are often placed in schools with empty classrooms. Because there is no classroom 
bathroom, safe playground or lunch in classrooms, more time is spent in transitions. The more 
transitions, the more behavioral control teachers have to assert. We also have to take into account 
the context of children’s lives. Is the classroom serving a large number of dual language speakers? 
Do children come from communities where there is a lot of violence? These circumstances should 
affect the way the classroom is organized. (Farran interview, 11/25/2020) 


The following themes, chapter excerpts from the Getting it Right full publication, and related questions are provided 


to stimulate additional engagement around the ideas and inquiry shared in this chapter. 
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> 1. As in years past, higher-income families were more likely than lower-income families 
to enroll their children in center-based care. Children from higher-income families often attend privately 
operated center-based child care programs, while children from lower-income families are likely to be enrolled in 


publicly funded programs such as Head Start and, more recently, state-funded prekindergarten programs (McFarland 
et al., 2017). 


One consequence of this division is that segregation of experiences by income begins in preschool. Moreover, privately 
and publicly funded programs have very different expectations and regulations. The fundamental motivation for the two 
sets of programs differs as well: private child care programs are more concerned with “care” and being of service to 
parents, while the public programs are more concerned with compensatory education to remediate presumed deficits 

in children’s preparation for school. This desire to offer compensatory education can lead to a greater emphasis on 
academic preparation and to more prekindergarten programs in public schools. An academic emphasis can have 
the unfortunate consequence of increased reliance on the sort of didactic instruction that may not lead to long-term 
child success (Lipsey, Farran, & Durkin, 2018) (p. 91). 


Right now, states are putting programs into place without a lot of guidance. As a field, we need a 
unified, neutral effort to pull together researchers and practitioners to pull up their sleeves to define 
the foundations of quality and develop a compendium of tools to use with specific populations of 
children. It will take looking at what data says is important in classrooms for children. We will have 
to learn to compromise with each other. (Farran interview, 11/25/20) 


Discussion Questions 


Research Design and Methods | Program Evaluation 


1. Explain what is meant by the compensatory education purpose for early childhood education 
(ECE). What does this purpose emphasize as the most important goals and features of ECE 


programs? 


* How does this frame view the locus of the “problem,” as inside the 
child or within the environmental and societal context in which the 


children (especially children of color) find themselves? 


¢ What evidence is there that the compensatory education purpose is 


necessary for some groups of children and not others? 
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¢ What is the evidence that compensatory educational practices also 


promote developmental and remedial goals? 


* Has the field changed its thinking about various purposes and outcomes 


for ECE? How have the research questions changed over time? 


2. How can various implementation research approaches to ECE incorporate a strength-based 


approach versus a deficit-only frame of children and families? 


3. What are possible obstacles in scaling these small, early-model programs today, apart from 


the ones listed in the publication? 


¢ What obstacles have been encountered in scaling many publicly 


funded programs? Are they the same obstacles? 


> 2. The work reported by my colleagues and | (Farran, Meador, Christopher, Nesbitt, 

& Bilbrey, 2017) is the result of my four-year partnership among myself, a group of researchers 
in the Peabody Research Institute at Vanderbilt University, and the Metro Nashville Public 
Schools. This work derived from an observation system developed for research purposes in the 1990s (Farran, 
Silveri, & Culp, 1991). Highly trained and reliable observers remained in classrooms for a full day, taking 

data throughout the day, several times a year. The system yielded important information about practices that 
mattered most for young children’s growth over the year and even into kindergarten and first grade. The practices 
determined to be important for children’s growth over the preschool year came to be called “the Magic 8” by teachers 
and coaches in the school system. The appendix contains an example of how one of the practices, reducing transitions, 


was translated into a tool for coaches to use in our continuing partnership with the district (p. 95-96). 


I’ve been interested in classroom observation since my graduate school days. | have learned 
that people are bad at rating things. As human beings, we just are. It is hard to use rating systems 
reliably. Most of us revert to our own judgements about what makes a good classroom. The 
other thing that happens with ratings is that if you ask one observer to look around a classroom 
and make a judgment as to how engaged children are, they tend to miss children who are quiet 
and unengaged. It is hard for human beings to make a judgment about how engaged groups 
of children are. (Farran interview, 11/25/2020) 
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Discussion Questions 


Research Design and Methods | Program Evaluation 
1. What ways do researchers typically observe, document, and rate instructional practices in 
center-based and home-based settings? 
¢ What are the limitations of these approaches? 
* How might these limitations be addressed? 
* How might the researcher's perspective and bias influence the work? 
2. How does the method of behavioral counts, as compared to behavioral ratings, change how 


the field understands classroom quality? 


* How does it change the measurement and assessment of structural, 


process, and instructional quality? 


3. Can research that uses behavioral ratings as a measure of quality be compared with research 


that uses behavioral counts? 


¢ In what ways are these bodies of work similar? 
¢ In what ways are these bodies of work dissimilar? 
4. What might be the benefits and drawbacks of including both measures of quality 


(counts and ratings) in your research? 


5. What are some advantages and disadvantages of using a method of counting 


teacher behaviors? 


* How might different professional roles or stakeholders (e.g., researchers, 
practitioners, coaches, program administrators, parents, children) vary in 


their response to this question? 
6. What are some advantages and disadvantages of using a method of counting student 


behaviors? 


7. How can interpretation and dissemination of behavioral count data help practitioners inform 


their practice? 
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> 3. Four areas among the eight—teachers’ listening to children, quality of instruction, positive 
climate, and child engagement—have also been investigated and found promising in several 
other studies. 


Teachers’ listening to children matters more than their talking to them. Language development and specifically 
vocabulary, has been one of the hardest areas to improve in early childhood classrooms. In general, however, few links 
have been found between teacher talk and child outcomes. Our research has shown that the amount of time teachers 


spent listening to children was actually the stronger predictor of children’s growth (p. 96). 


The teacher’s quality of instruction is as important as the student's acquisition of basic skills. “Productive 
conversations,” especially teachers’ asking questions and listening to children’s answers, are components of a more 
general factor related to the quality of instruction. In a recent book, William Gormley (2017) makes a persuasive 
argument that encouraging critical thinking through inferential teacher-student interactions may be one of the most 
important experiences in helping children be successful. He also argues that children from disadvantaged backgrounds 


are less likely to have these kinds of experiences (p. 97). 


Positive classroom climates promote learning, and the importance of a positive learning environment cannot be 
overestimated, especially for young, vulnerable children who may be having their first educational experience 

in a formal setting. The classroom climate is particularly important for atrisk children, who typically have had a 
higher than average number of adverse childhood experiences. To promote resiliency in such children, the classroom 
must promote a sense of belonging, with caring and nurturing adults (Sciaraffa, Zeanah, & Zeanah, 2017). A highly 
negative classroom can actually function as an additional adverse experience, contributing to rather than buffering the 


cumulative stress that results in long-term negative health and social outcomes (p. 99). 


Children’s active engagement in learning is key, and engagement should not be confused with compliance. 
Children can be quiet and nondisruptive without being engaged. When children are actively involved in learning, they 
can be noisy (in a productive way). When young children are engaged, they are excited and highly attentive to the 
learning activity. Engagement is intertwined with all the other components described so far. For example, the level of 
positive emotional support in a classroom predicted children’s level of classroom engagement (Castro, Granlund, & 
Almavist, 2017) (p. 100). 
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Policies always look for the silver bullet. We try to make things easier than they are as in, 
“Let’s make teachers teach to a scripted skill-based curriculum.” There are no silver bullets. I’m 
not saying we don’t need a curriculum. But that the teacher's interaction patterns matter. And that 
we need to see teachers as thinkers. One size does not fit all when diverse teachers are working 


with diverse children. (Farran interview, 11/25/2020) 


Discussion Questions: 


Research Design and Methods | Program Evaluation 


1. How do measures of structural quality compare with measures of 


process quality? 


* How might the results be interpreted and implemented 
differently? 


2. What factors do you think lead the field to continue to focus on 
structural features of ECE quality that potentially have weak or small 
associations with child outcomes vs. identifying specific instructional 


approaches that promote children’s learning? 


3. What additional practices does research indicate could be added to 


“the Magic 8” to be more closely tied to child outcomes? 


4. What research questions remain for how and why listening to children, 
quality of instruction, positive climate, and child engagement seem to be 


effective? 
¢ What questions remain for other instructional approaches? 


¢ What questions remain for different groups of children? 


5. How can factors of race, ethnicity, socioeconomic status, and 
neighborhood characteristics influence the way the four factors 


explained in this chapter are measured? 


“The Magic 8” 


1. Reduce time spent in transition. 
2. Improve level of instruction. 
3. Create a positive climate. 


4. Increase time teachers listen 
to children. 


5. Plan sequential activities. 


6. Promote cooperative interactions 
between children. 


7. Foster high levels of child 
involvement. 


8. Provide math opportunities. 


Source: Farran, D.C., Meador, D., Christopher, 
C., Nesbitt, K.T. and Bilbrey, L.E. (2017), 
Dato-driven improvement in prekindergarten 
classrooms: Report from a partnership in an 
urban district. Child Development, 88: 1466- 
1479. https: //doi.org/10.1111 /cdev.12906 


6. How can “the Magic 8” improve early educator preparation programs and professional 


development systems to better support the ECE workforce in creating supportive, stimulating 


learning environments for young children? 


¢ What roles can research play in this? 
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> Moving forward. 


“We have to decide what it is we want to know about classrooms, and it may be different depending 
on the children who are learning there. For example, if there are a high number of DLLs, we should 
come in with a tool that looks at kids who are learning languages. The same with children who come 
from communities of high violence who might benefit from a tool that focuses on creating a safe, 


accepting, warm environment.” (Farran interview, 11/25/2020) 


Discussion Questions 


Research Design and Methods | Research Collaboration and Partnership 


1. Brainstorm how evaluative tools should be responsive to children’s diverse needs and identify 


enriching and supportive instructional practice. 


2. How will genuine collaboration between researchers, policymakers, and practitioners 


contribute to the development of such tools? 


¢ What do you bring to the table that is needed and why? 
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IMPROVING QUALITY AND IMPACT 
THROUGH WORKFORCE DEVELOPMENT 
AND IMPLEMENTATION SYSTEMS 


Robert C. Pianta, Ph.D., University of Virginia 
Bridget K. Hamre, Ph.D., Teachstone 
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In Improving Quality and Impact Through Workforce Development and Implementation 
Systems, Robert C. Pianta and Bridget K. Hamre point to the need for systematic improvements 
in professional development (PD) systems to provide children with effective education across 
early childhood settings. Though professional development is widely used as a strategy to 
improve child outcomes, it is hampered by varying standards across states, less than effective 
coaches, and gaps between how implementation science says it should work and how it is 
practiced. Professional development provided with greater intention and integration is more 
effective and offers a unified quality experience for children across settings and teachers. 


The summary above appears in the Getting it Right Chapter Summaries resource. 


We want folks out there in the field to use evidence to guide their decisions. Yet, evidence-based 
is not a pill you can take to improve outcomes. Educators are told to adopt evidence-based curriculum 
and assessments, for example. Evidence tells people that in general, under certain circumstances, 
this outcome is more likely. It tells us about a general tendency. But we can’t leave them there. 
We have to take them further so that they are saying, “I need to think about this evidence and if 
and how it applies to me and whether I can replicate conditions that | am reading about in this brief.” 
(Pianta interview, 11/9/2020) 


The following themes, chapter excerpts from the Getting it Right full publication, and related questions are provided 


to stimulate additional engagement around the ideas and inquiry shared in this chapter. 


> 1. Recognizing the need and value for PD, policymakers have made significant investments 

in the workforce, which is a first step. But that investment does not focus enough on proven- 
effective PD models. Unfortunately, teachers rarely experience PD that reflects features of specificity and alignment 
to practice. In fact, a recent survey that was representative of the 1 million teachers in center-based programs for 
children aged 0 to 5 years indicates that the predominant form of PD is a one-hour workshop only tangentially 
connected to teachers’ everyday practice and known to be ineffective (McCormick Center for Early Childhood 
Leadership, 2016; Zaslow, Tout, Halle, Whittaker, & Lavelle, 2010) (p. 113). 
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Discussion Questions 


Research Design and Methods 


1. What challenges do a fragmented early childhood education (ECE) system and corresponding 
professional development system pose for researchers interested in investigating effective 


professional development models and systems? 


2. What types of research design, research questions, sampling procedures, analysis methods, 
and results interpretation are needed to establish an effective PD model? 
* How does correlation vs. causation play a role? 
3. How can an implementation research approach be used to examine the ways early educator 
preparation can positively change instructional practice and, in turn, promote child outcomes? 


4. How can implementation research examine professional development systems? 


5. What types of research studies can be designed to evaluate the interaction of preparation and 


PD systems in improving instructional practice? 


6. Chapter 5 explains that there is little agreement about competencies and qualifications for 
early educators across different ECE systems and states. What kinds of problems does this lack 


of consensus pose for early educator preparation and the PD system? 


7. How can research examining relationships between early educator preparation, professional 
development, and instructional practice account for the different competencies and 


qualifications in the field? 
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> 2. Implementation science can offer a framework for knitting together the potential of 
proven-effective training and PD with the everyday realities of classroom practice, program 
capacity, and surrounding systems. This is because implementation science, with its focus on identifying and 
engineering the conditions that influence and explain strong and weak implementation, can create the kind of systemic 
and aligned programs of professional training and development that foster improvements in classrooms and impacts 
on children (p. 114). 


We have made ad hoc arrangements around curriculum, instruction, and data. But now we've moved 
the needle. We are entering a new frontier with implementation science and improvement science. 
We know “X” works. How does it get employed? What are the decision scaffolds? There is now a 

science to help us answer those questions. (Pianta interview, 11/9/2020) 


Discussion Questions 


Research Design and Methods | Program Evaluation 
1. What research measures and tools are needed in professional development systems to help 
early educators improve their practice? 
2. How can changes in practice be best assessed? 
* How does diversity in students and teachers influence the choice of 
assessments and their use? 


* How do different places, locations, and circumstances influence these 


assessments and their use? 


3. How should findings from research measures, tools, and supports be disseminated so that 
they apply to practitioners (e.g., early educators and leaders) in the field and so that lessons 


learned can be applied in practice? 
4. How should these findings be disseminated to policymakers and other stakeholders? 
5. How can implementation research contribute to improving the quality of educator PD? 


¢ What are some of the questions that can be studied? 
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6. What are examples of implementation research studies that can help shape PD approaches 
and content that are responsive to the diverse needs of early educators aiming to improve 


their practice? 


* How might these studies change depending on your state or locality, 


ECE setting, and age of children served? 


7. How do the systems that surround ECE programs interact with these educator preparation and 
PD initiatives? 


* How do these systems interact with research? 


> 3. Reports have clearly described the features of PD that relate to improved practice 
and student learning (Zaslow et al., 2010). When targeted, practice-aligned PD supports are available 
to teachers, student skill gains can be considerable—at times on the order of half a standard deviation and higher in 


some subgroups. 


Focus on teacher skills and relevant knowledge. A starting point for identifying, implementing, and eventually scaling 
effective PD is to consider the PD target and the system in which it will be implemented. As Burchinal (this volume) 
suggests, classroom observation of teacher practice is often viewed as a source of information on the focus or target of 


PD, as is teachers’ knowledge of children’s development or of a curriculum. 


It may seem obvious that PD should focus on evidence-based teaching practices, but experience and the limited 
available data suggest that much PD for teachers does not do so. In one review of 256 published studies of ECE PD, 
only 25% had explicitly focused on teaching practices (Snyder et al., 2012). And the vast majority of practice-focused 


PD targets more generalized teaching practices, early literacy, and/or social-emotional teaching (Schachter, 2015). 


Ensure sufficient intensity and duration. Intensity and a greater duration of PD consistently leads to improvements in 
teachers’ practice (Garet, Porter, Desimone, Birman, & Suk Yoon, 2001; Markussen-Brown et al., 2017). Markussen- 
Brown and colleagues (2017) reported a wide range of intensity among the studies they included in their meta-analysis 
of PD, from six to 450 total hours; they found greater changes in teaching practice among PD programs with greater 
intensity. Unfortunately, we do not know exactly how much PD is enough, though it is likely that the answer depends 


greatly on the desired outcome. 
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In sum, ample evidence from rigorous experimental studies shows that PD focused on teacher practices or relevant 


knowledge can improve the quality of teachers’ skill and, to a lesser extent, children’s learning. We have curricula, 


methods of practice, and tools that can predictably improve teachers’ knowledge and skill, and a number of them also 


show evidence of further benefits for children’s learning. At the same time, there is fairly broad agreement that PD for 


ECE teachers as typically implemented by states and school systems throughout the country is not all that effective. 


The opportunity to deploy PD investments for greater impact holds tremendous promise for improving the benefits of 


programs for children (p. 115-118). 
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Discussion Questions 


Research Design and Methods | Program Evaluation 
1. How can implementation research support the alignment of early education preparation, 


professional development, and instructional practice? 


2. Are there benefits and challenges for building a PD system that spans the continuum of birth 
through third grade? 
¢ What role can researchers play in such an effort? 


¢ Who else should be engaged in this effort, and what benefits could 


such a collaboration bring? 
3. Choose a feature of a PD system. How would you design an implementation study to assess its 


effectiveness in enhancing the practice of early educators and improving child outcomes? 


4. How might the design of a study differ when measuring a PD system that evaluates teacher 


practices in the classroom, as compared to teacher's general knowledge? 


¢ What differences would there be in the interpretation and dissemination 


of these results2 


5. What factors contribute to the success of high-fidelity implementation of PD and the successful 


scaling up of such PD2 


¢ What role can research play? 
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> 4. To improve the quality and impact of programs at scale through workforce development, 
we must explicitly specify the enabling architecture—the incentives, standards, training and 
implementation protocols, quality control procedures, and certifications that shape the actions 
of various people in the system (teachers, purveyors, programs) to produce high effort and 
focused participation. All too often, these components of a workforce development system are misaligned with 
one another, with the needs of the workforce, and with the support structures needed to deliver the types of proven- 
effective PD described here (p. 118). 


The only way this work can be done is by researchers in collaboration with practitioners working 
as partners for a bigger purpose... Very often the translation of research is conducted by the person 
who generated the original piece. But how can you translate without seeing how it lands? 
(Pianta interview, 11/9/2020) 


Next, we identify several conditions that are key to closing the gaps between PD that has been proven effective 


under local or controlled conditions to implementation with benefits at scale (p. 120). 


Use a clear and focused PD program or model. Zaslow and colleagues (2010) have described the features of 


effective PD programs, which include a focus on: 


a) students’ skill targets and developmental progressions (e.g., developmental progressions in decoding 
skills); b) improving teachers’ skillful use of instructional and social interactions to promote student 
engagement and learning (e.g., feedback or conversation); and c) fostering teachers’ skills and knowledge 
to effectively implement curricula and appropriately engage children with content (e.g., delivering an 


effective and engaging activity on teaching cardinality). 


Provide necessary supports for the PD workforce. PD’s success depends in large part on the people who train and 
coach teachers. This means hiring, training, and supporting the PD workforce. But little research has examined these 
elements of program delivery, and many evidence-based PD models fail to provide much detail about them. Among 
evidence-based PD models that do provide such detail, this workforce typically consists of experienced ECE teachers, 
often with master’s degrees, who have relatively extensive training and ongoing support in the particular PD model 
(McCollum, Hemmeter, & Hsieh, 2011; Piasta et al., 2012; Powell, Diamond, Burchinal, & Koehler, 2010). Lloyd & 
Modlin (2012), reporting on how they delivered three coaching models in Head Start programs, suggest that successful 
coaches have three maior attributes: knowledge of the coaching model, general coaching and consultation skills, and 


knowledge of early childhood development and teaching. 
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Use data to target and improve PD. However, data can not only help to focus PD but can also track its implementation 
and success. Lloyd & Modlin (2012) describe a simple but effective method for supporting the coaching delivered as a 
part of the Head Start CARES project. They use brief online surveys, logs, and fidelity reports to help support technical 
assistance and management in their monitoring of coaching implementation. Similar systems are provided with the 
scaled-up version of MTP [MyTeachingPartner] (Early et al., 2017). Even the simplest information, such as logs of the 
frequency of contacts between teachers and coaches, can be powerful ways to improve the intensity of coaching if 
they are used to monitor coaches’ efforts and provide feedback. ... As states build systems of PD support online and link 
them to various forms of credentialing (including micro-credentialing), the result can be more fully integrated alignment 
of teachers’ PD needs and goals, PD inputs to teachers, supports for effective delivery (by coaches, instructors, or web 


systems), and structures that codify and encourage teachers’ participation and progress. 


Link workforce development systems and incentive structures. However, registry systems are being developed 
that codify individual teachers’ records of acquired PD (National Registry Alliance, 2013a) and perhaps even the 
competencies they attain, which will mean greater capability to identify and encourage effective PD as well to tie those 


experiences to accrued competence and certifications. 


Certify PD providers. The skills and impact of those who provide PD support to teachers and programs vary widely 
(Soliday-Hong, Walters, & Mintz, 2011), and there are very few systems for documenting their expertise and 
effectiveness. Although almost half of the states have developed tracking systems for PD providers (Institute of Medicine 
and National Research Council, 2015), none have effectiveness metrics or standard certifications and training. Some 


have moved beyond tracking to comprehensive training and certification requirements for providers. 


In some states, PD providers must register and complete training (National Registry Alliance, 2013b), but these systems 
are typically voluntary and their requirements are not particularly stringent. Clearly, PD providers and coaches need 


more intensive training and certification programs (p. 121-123). 
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Discussion Questions 


Research Design and Methods | Program Evaluation 


1. A clear and focused professional development model is the first condition listed for 
implementing PD with impact. Why is it important that curricula, assessment, and other 
enabling supports are aligned? 


* How can research be used to ensure this alignment? 


2. It is noted that very few programs use a formal manual or set of materials to guide coaches as 
supports for the ECE workforce. How might you design a research study to examine supports 


that foster effective coaching practices? 
* How could you disseminate the results in a way that might change 
policy and practice? 


¢ Should coaching practices be the same across different types of ECE 


programs, settings, and children served, etc.? 
* If not, what kinds of differences might you expect? 
3. What are the benefits and potential drawbacks of course-based PD in higher education? 


4. What types of data are required for continuous improvement of PD at a federal, state, and 


local level? 
¢ What would each contribute to strengthening the PD system and 
thereby improving child outcomes? 
¢ What are the potential challenges to gathering and using this data? 
¢ What are potential strategies to overcome these challenges? 
5. Design a workforce development system, including incentives that could help encourage 
effective PD. What are the research questions that arise regarding implementation? 
¢ How could you measure the fidelity of this system? What fidelity issues 
might arise? 


6. Considering the factors that allow for the successful implementation of PD programs, what 


research questions remain? 
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> Moving forward. 


56 


“Any translated research product has to be framed and written and content exposed from the 
perspective of the person who is reading it. As researchers, we have fo figure out a way that the 
translation reflects the complexity of the issue. For example, what you have evidence for and don’t have 
evidence for. We have to respect both the complexity of the science and respect the practitioners’ ability 
to understand and use findings in their decision-making and work...Ilmplementation science can provide 


the young researcher with an intellectual home.” (Pianta interview, 11/9/2020) 


Discussion Questions 


Research Approach and Stance 
1. As you look forward, what steps do you think you, as a researcher, must take to ensure that 
research findings on PD are translated in ways that are understood and used by the field? 
¢ What steps does the field need to take in general? 


2. What role can you see yourself, as a researcher, playing in building a connection between 
what we know from research about effective PD and what we do as a field to support 


professional learning and development? 
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In Addressing Equity in the ECE Classroom: Equal Access and High Quality for Dual Language 
Learners, Linda M. Espinosa discusses research outlining the benefits of early bilingualism. 
She also presents strategies that all early childhood education (ECE) teachers can implement 
to support dual language learners’ (DLL) improved outcomes through the acquisition of 
English, while also maintaining their home language. Directions for future implementation 
research are provided to help fully understand factors that influence early bilingualism, the 
attendant cognitive, linguistic, and social advantages, and effective practices for instructing 
and assessing DLLs. 


The summary above appears in the Getting it Right Chapter Summaries resource. 


The study of bilingual children is almost its own field of study. Who are these children in 
the U.S.? How are decisions about how and what to teach them and how to measure their progress 
impacted by the fact that they do not speak English at home and may have a varied culture? 
(Espinosa interview, 11/24/2020) 


The following themes, chapter excerpts from the Getting it Right full publication, and related questions are provided 


to stimulate additional engagement around the ideas and inquiry shared in this chapter. 


> 1. Historically, most research examining the growth, progress, and achievement of DLLs has 
focused on differences between DLLs and non-DLLs, judging DLLs’ performance using norms 
designed for English-only populations without considerations for the unique linguistic and 
developmental trajectories of children whose first language is not English.' This approach has 
often led to a “deficit perspective” that views DLLs as having less potential and fewer academic 
abilities than their monolingual English peers because of their lack of English proficiency. |n 

fact, policymakers have sometimes referred to “the extra burden” of having to learn two languages during the early 
years. The deficit perspective, however, often negatively affects teachers’ views of DLLs’ potential, and it is, moreover, 


contradicted by current research (p. 135). 


The scientific consensus is that children who become fully proficient in both their home language and English are likely 
to reap benefits in cognitive, social, academic, and professional outcomes and to be protected from brain decline at 
older ages (NASEM, 2017). This suggests we should view the development of DLLs through the powerful advantages 
of having more than one language. The assets associated with bilingualism and biliteracy have been well documented 


and should be recognized and supported (p. 135). 
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I think the discrepancy between what academics know and what practitioners implement is huge. 
There can even be a discrepancy between what some practitioners know and do. People have 
distorted views of how successful these children are. Too few see these children as advantaged 
because they speak more than one language. In fact, our achievement data shows them behind 
because they have been improperly assessed and understood. (Espinosa interview, 11/24/2020) 


Discussion Questions 


Research Design and Methods | Research Approach and Stance 
1. What does it mean to take a strength-based vs. deficit-based research approach? 
¢ What are some examples of what each looks like in various ECE 


settings (e.g., an infant/toddler setting, a program serving children with 


special needs, and a program with DLLs)? 


¢ In what ways can the approach assumed by researchers shape 


findings, the communication of findings, and the impact on a program? 


¢ What knowledge is required to utilize a strength-based, asset-based approach 


in research studies? What shifts are required in a researcher's stance? 


2. Why do you think deficitbased assumptions about the development and learning of 


dual language learners persist in research and practice? 


How might these assumptions have an impact on practitioners’ 


instruction and support of DLLs? 


How might these assumptions have an impact on children’s experience 


of school and learning? 


How do such considerations influence research related to DLLs and to 
the instruction of DLLs? 


How can you frame your research from an asset-based perspective? 


How can teachers frame their instruction with that same asset-based perspective? 


3. What additional research is needed to help move the field away from this deficit-based 
perspective that is so common in research and instructional practices and more toward an 


asset-based perspective that focuses on the strengths of DLLs? 


¢ What are different ways families can play a role? 
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4. On page 134, Espinosa identifies a wide range of questions that researchers and practitioners 
are asking to address this challenge. What questions would you choose to study and why? 


What would your hypotheses be? 


5. What research questions are needed to further understand how to close the substantial and 


persistent achievement gap between DLLs and native English speakers? 


¢ What are the strengths and resilience factors that also need to be 


included in any study of DLLs? 
* How would you compare the two groups in a proposed study? 
¢ What contextual factors would you consider? 
¢ What types of assessments would you use? 


¢ What are some research designs that could offer insights into this gap 


and suggest new teaching practices that could close it? 


6. What research questions are needed to further understand how to support children’s bilingual 


or multilingual language development? 


7. How can ECE programs and specific instructional practices better support children’s language 


development so they can become bilingual or multilingual? 


> 2. A 2017 report by the National Academy of Sciences, Engineering, and Medicine (NASEM), 
Fostering the Educational Success of Children and Youth Learning English, offers a research 
synthesis on the development and achievement of DLLs from birth to age 21. This consensus 

study has yielded a comprehensive view on language development, school practices, and educational policies that 
impact DLLs’ growth and school success. It reports four major interrelated conclusions that are central to improving the 
educational outcomes for DLLs. First, all children are capable of learning more than one language from the earliest 
months of life and benefit from early exposure to multiple languages. Second, high levels of proficiency in both the 
home language and English are linked to the best academic and social outcomes. Third, the earlier a child is exposed 
to a second language, the greater their chances for full bilingualism. Fourth, home language loss is currently the norm 
for DLLs, particularly once they enter English-speaking ECE settings, which undermine the possibility of full bilingualism 
and may place the child at risk for unhealthy family relations, including estrangement from their cultural heritage 


(p. 138-139). 
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Because most in our field do not have a background in dual language children, people have defaulted 
to simplistic ideas of how these children are different and how to determine their accomplishments. 
We are making progress. For example, we've designed an in-depth family interview to learn about 

a child’s exposure to English. But there is so much more to be done... | often tell people, “Make a 
tentative hypothesis about why a child is or isn’t performing and ‘keep the doors open’ as you check 
into what might be reasons why. Language development can be uneven—even for kids who speak 
one language.” (Espinosa interview, 11/25/2020) 


Programs with three, four, five languages usually end up just teaching children English. They say, 
“We can’t bring in all these languages.” It’s just not true. There are methods to address cultural and 
language diversity. Most people in our field don’t have backgrounds in dual language children, so 
they think either dual language (hopefully) or teach only in English. (Espinosa interview, 11/24/2020) 


Discussion Questions 


Research Design and Methods | Program Evaluation | Research Approach and Stance 


1. As you review the Summary of Findings of the NASEM report (2017), what findings are in 
line with what you knew or expected? Please reference page 66 of this guide for The Summary 


of Findings. 
¢ Which of these findings might have been similar or different from your 
expectations? 


¢ What research questions remain that would have been beneficial for 


practitioners and policymakers? 

2. Consider the extensive summary of research findings on page 138. How might these footnotes 
be useful or not for practitioners and policymakers to identify ways to better support and 
instruct DLLs? 

¢ How could researchers best disseminate these findings? 


3. What research questions are needed to help practitioners integrate DLLs’ home languages and 


the needs and wants of families into classroom practices? 
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4, What research is needed to help practitioners support DLLs on an individual level? Consider 
that the learning environments often include children who speak a variety of home languages, 


all of which may be different from the teacher's home language. 


¢ What could be the benefits and challenges of having a teacher who 


comes from a background similar to DLLs in the classroom? 


5. What approaches and strategies might researchers who speak one language use to ensure 
the development of an effective research process and design when looking at supporting 


children’s home and second languages? 


6. What factors should be considered in implementation research related to examining how 


instructional practice influences the bilingual and multilingual development of young children? 


¢ What strategies might researchers, policymakers, and practitioners use 
to partner with DLLs’ families to promote and support these practices at 


home and in the classroom? 


Summary of Findings of NASEM (2017) Report for DLLs 0-5. 


The major findings about DLLs ages birth to five from the NASEM (2017) © DLLs should be supported in maintaining their home language in 


report include the following: preschool and early school years while they are learning English in order 
© All young children, if given adequate exposure to two languages, can to achieve full proficiency in both languages; 

acquire full competence in both languages; ¢ DLLs’ language development is enhanced when adults provide frequent, 
* Early bilingualism confers benefits such as improved academic outcomes responsive, varied language interactions that include a rich array of 


in school as well as enhancement of certain cognitive skills such os diverse words and sentence types. For most DLL families this means they 


executive functioning: should continue to use their home language in everyday interactions, 


: storytelling, songs, and book readings; 
* Early exposure to a second language—before three years of age—is 


related to better language skills in second language, English; © There is wide variation in the language competency among DLLs that is 
due to multiple social and cultural factors such as parents’ immigration 


Tne Me ge teeta ean DAP Sena om status and number of years in U.S., family Socio-Economic Status (SES), 


monolingual children: they may take longer to learn some aspects : 4 
; ; status of home language in the community, resources and amount of 
of language that differ between the two languages and their level of 


support and for both languages. 
proficiency reflects variations of amount and quality of language input; id eee 


¢ The cognitive, cultural, and economic benefits of bilingualism are tied Source: NASEM (2017). Promoting the educational success of children and youth learning English: 
to high levels of competence including listening, speaking, reading, and —_Yomising futures. The National Academies Press. 


writing in both languages, e.g., balanced bilingualism at kindergarten 
entry predicts best long-term outcomes; 
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> 3. It is important for educators to recognize that there are differences between DLLs and 
monolinguals. Preschool DLLs seem to show a different pattern of strengths and needs than monolinguals. They 

are at risk for low levels of oral language development if they don’t receive frequent high-quality enriched language 
learning opportunities in both languages. Their basic mathematical understandings may differ from those of English 
speakers if their first language uses different language constructs for expressing math concepts such as counting, plurals, 
grouping, and so forth. They may also excel in certain executive function skills such as cognitive control, and they often 
demonstrate social-emotional strengths (NASEM, 2017) (p. 140). 


To provide equitable early education to linguistically diverse children, ECE teachers must consistently implement a 
set of instructional adaptations across multiple settings. One core necessity here is to recognize that these children 
are learning content or conceptual knowledge at the same time that they are also learning the language in which that 
content or concept is expressed. Thus, instructional approaches that focus on monolingual English speakers need to be 
adapted and enhanced (Castro, Espinosa, & Paez, 2011; NASEM, 2017) to build on what children already know in 
their first language while they are also adding English (p. 133). 


Implications of research for instructional practices for DLLs. Before teachers can specifically address instructional 
goals and strategies for DLLs, they must first get to know the children. They need to gather formal and informal 
information on their students’ backgrounds and their early language learning experiences as well as abilities, including 
how much exposure they have to both the home language and English and how much they use each. During face-to- 
face interviews with parents, teachers can learn about family values, language preferences, cultural traditions, and the 


ability to partner actively with teachers in the classroom (p. 142-143). 


Although common features of high-quality early education described throughout this volume are beneficial for 
all children, DLLs require additional instructional support. The NASEM report (2017) outlines a number of instructional 
strategies and enhancements that have been linked to improved achievement for DLLs in early education settings 


(p. 143). 


Currently, few states require ECE teachers who work with young DLLs to have specialized training or coursework 
focused on meeting the needs of such children and their families (Espinosa & Calderon, 2015). The NASEM (2017) 
report concludes, “The educator workforce, including early care and education providers, educational administrators, 
and teachers, is inadequately prepared during preservice training to promote desired educational outcomes for dual 


language learners” (p. 462) (p. 145). 


Unless you believe “in your bones” that having a second language in addition to English is a gift, and not a 
disadvantage, and diversity is a resource, not a problem to be solved, you are likely to respond to DLL children in ways 
that discourage the continued use of the home language, especially if you are not fluent in the child’s home language 
(Espinosa & Magruder, 2015, p. 80) (p. 142). 
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Discussion Questions 


Research Design and Methods | Program Evaluation | Research Approach and Stance 
1. What additional research is needed to better understand how practitioners can best support 
and instruct DLLs? 


2. What are realistic expectations for including every child’s language in some classrooms? 


3. What are some of the differences that may exist in DLLs’ and monolinguals’ learning and 
development? Consider factors such as home language prominence, instructional content and 
practice, family culture and values, and school culture and values. 

* How do such considerations influence your research priorities and approach? 
* How do such considerations influence the interpretation of your research results? 


4. What are the assessment and measurement considerations in research related to DLLs’ 
development, including learning of specific skills, such as language and literacy, math, and 


other knowledge concepts? 


¢ What is the best way to assess DLLs? How might your data collection team and the 
tools available to you as a researcher influence the practicality of your previous 


response? 
5. How might a practitioner's assumptions influence whether instructional content and strategies 


to foster bilingual and multilingual development are integrated into a program's daily life? 


6. What research methods might a researcher use to understand or measure the impact of their 


assumptions? 


7. As you read core content elements (p. 145) and competencies (p. 146), what research 


questions would you ask regarding the preparation and skills needed to work with DLLs? 


¢ Which of these questions would you research first? Why and how? 


8. In what ways can ECE and PD systems be enriched to include the content, elements, and 


mindset necessary for educators to provide an equitable education to DLLs? 


* In doing so, what might the unintended implications (positive and negative) be for 


other monolingual children in the classroom? 
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9. What implementation research approaches may help practitioners learn about and understand 


the family context of DLLs? 


¢ What can be learned about a family’s culture, values, and views of their home 


language and English? 


¢ How might they influence interactions with a child? 


> Moving forward. 


Substantial research has been done on the capacity of all children to successfully become bilingual, the factors 
that influence early bilingualism, and the attendant cognitive, linguistic, and social advantages, and there is also an 


emerging scholarship on effective practices for DLLs. Yet there are still many gaps in our knowledge (p. 147). 


Discussion Questions 


Research Collaboration and Partnership 
1. As you read the questions posed by Espinosa on pages 147-148, choose one that you would 
like to study with a research team. 
¢ Why is this your choice? 
¢ What would your hypotheses look like? 
¢ What role do you see yourself playing on the research team? 


¢ Who would you engage as collaborative partners on this research team, and how 


would they benefit the project? 
¢ What insights, information, and experience will inform your contribution to the work? 
¢ What research approaches will be most helpful? Why? 
¢ What practical or logistical considerations may arise during this project? 


* How would you work around them? 
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In Vignette: Building a High-Quality Program—the Boston Public Schools Experience, Jason 
Sachs, who established and continues to lead the expansion of the Boston Public Schools 
system’s Prekindergarten-2nd grade program, relates his and his staffs’ experience in building 
an equitable, high-quality early childhood education (ECE) system that produces measurable 
outcomes. Sachs talks about the keys to success: committed city leadership; a focus on the child; 
resourcing staff, principals, teachers, and paraprofessionals to do their best work; developing 
strategic plans; and using data for evaluation that feeds continuous improvement. 


The summary above appears in the Getting it Right Chapter Summaries resource. 


Sachs offers a window into building and scaling up the Boston Public Schools’ prekindergarten to second grade 
program. He describes the intentional use of research to guide change and the realities encountered while conducting 
implementation research (p. 8). The experience of the BPS Department of Early Childhood grounds this volume by 
presenting the initiative’s context and the “real work” of practitioners and researchers collaborating over the last 14 
years on a path to achieve their mission. Supporting his work—and that of others in the field working day by day to 


create stronger outcomes for all children—is the purpose of implementation research. 


The three activities below are designed to help students consider key elements of effective implementation research 
as they begin to intentionally determine their stance as researchers: consideration of context, development of genuine 


research collaboration, and the need for and development of tools to assess fidelity. 


The first activity focuses on the importance of context in designing and implementing an ECE program or system. The 
second invites students to consider the need for genuine collaboration and the “natural tensions” that can occur as 
researchers, practitioners, and policymakers strive to implement evidence-based, informed practice throughout all the 
phases of implementation research. Activity three leads students to explore the need for and development of fidelity 
tools to measure whether a program’s delivery matches the program’s original intended model and purpose. Each 
activity is designed to be thought-provoking and to challenge students to apply insights about implementation research 


to real-world situations. 


The following themes, chapter excerpts from the Getting it Right full publication, and related questions and activities are 


provided to stimulate additional engagement around the ideas and inquiry shared in this chapter. 


> 1. Context is key to continuous quality improvement efforts. At the opening of Chapter 7, Sachs 
invites readers to “become like the horse breaking free—taking what is useful for their own contexts.” He shares his hope 
that “this article will help your work as you set out to build or improve your own preschool systems and partner with 


your own public schools.” 
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Articulating a change theory, strategic plan and processes, and protocols help people put their egos 


aside as they build a collective will and interactions. People see where and how they can fit in. You 


know where you want to go and have a roadmap to get started on your way. (Sachs interview, 11/17/2020) 


Activity 


In small groups, think about how to “break free” in 
considering designs for ECE programs and using research 

to refine the implementation of an ECE program or system in 
your own local context. Choose a specific ECE program or 
system and consider contextual factors like program/system 
purpose, whether the program/system is being implemented 
as planned (why or why not), who is the program serving 
and how well, what implementation supports are in place or 
not, what the characteristics of the population implementing 
the program and those receiving the services are, and if data 
is available on program implementation outcomes and child 
outcomes. As part of your deliberation, also reflect on the 
key lessons learned by BPS (p. 165-172) as they worked 

to continuously improve the quality of their ECE program/ 
system. After your discussion, write down brief answers to the 


following questions: 


1. What refinements are needed, for what purpose, and for 


what outcomes? 


e¢ What is the rationale or evidence-base for such 


changes? 


Key Lessons Learned 


1. There are natural tensions in a 
research-practice partnership. 


2. Planning matters. 


3. What you don’t do is as 
important as what you do. 


4. Data helps you work smarter. 


5. It is important to create strategic 
plans, and to stick with them. 


6. The curriculum needs to keep 
pace with the students. 


7. Use NAEYC accreditation as 
a driver to set quality at the 
school level. 


8. Whether degrees are critical 
for education workers is 
fraught issue. 


9. Creating a pre-K model for 
community-based programs 
is crucial. 


¢ What are the contextual factors that necessitate, support, or hinder such changes? 


2. How would such changes be implemented? 


¢ What are the current constraints and how will they be addressed? 


3. How can research support these refinement efforts? 
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4. What are the research questions? 


¢ What is the appropriate timing for developing research questions given 


potential or new implementation of program/system changes? 


5. Which implementation research designs and approaches best fit the research questions? 


Afterward, share and compare the responses across the small groups. How are they similar or 


unique? Were there common themes? 


This same activity can be replicated for other ECE programs and systems outside of your local 
context. Compare and contrast the responses to these questions from your local context to those 
outside of your context. How are they similar or unique? Were there common themes? How does 
the varying context of these programs and systems contribute to the similarities or differences in 


the responses? 


> 2. Building research partnerships to support data-driven, evidence-based quality 
improvement efforts. An element of the success of BPS is Sachs’s long-term relationship and articulated 


understanding with a research partner. 


There are natural tensions in a research-practice partnership. Some questions are too academic in 
our department's view; that is, they might benefit the field but not the department. We turn down 
ideas that fall into this category if they represent a burden without benefit for the district. Conversely, 
sometimes the department has had a question or a “need to know” that is either not of interest to 
academics or not publishable. Weiland and her team have generally taken these on just the same. 
Their view is that to be good citizens and partners and to learn as much about the district as possible, 
it is important to address them. (Sachs interview, 11/17/20) 


The BPS Department of Early Childhood uses data for a variety of purposes, such as identifying systematic 
weaknesses across classrooms and targeting PD accordingly. For example, classroom quality data collected in 2010 
in prekindergarten and kindergarten revealed that although the program had the highest instructional quality of any 
large-scale prekindergarten to date (Weiland, Ulvestad, Sachs, & Yoshikawa, 2013), teachers were not doing enough 
to support children’s conceptual development. Professional development was then modified to target best practices 

in this area. We also created a teacher-friendly template that displayed each teacher's results compared to district 
averages and areas for growth. Coaches worked with teachers to help them understand the implications of their scores 


for their practices (p. 161). 
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I’ve learned to be a manager and how to connect people’s work to a strategic plan and help them 
see the connection. We've stayed with it for a long time so we could collect and use data to adjust as 
needed. It is the process of doing the work with a strategic plan and goals over time that lets us evolve... 
People need to learn management, strategic thinking, benchmarking skills. | have always 
had partnerships with people in different fields. (Sachs interview, 11/17/2020) 


Activity 


The building of researcher, policymaker, and practitioner partnerships to implement evidence- 
based, informed practice requires genuine collaboration throughout all the phases of 
implementation research. Research partnerships can be formal in nature according to various 
research-practice partnership (RPP) models, or they can be informal.' For this exercise, in small 
groups develop a logic model to outline the development of a formal RPP that is focused on 
examining questions related to ECE program implementation in your local context using a 
data-driven, practice-based approach. Attempt to add the perspective of colleagues such 

as policymakers, practitioners, and parents. Include roles and responsibilities for partners, 
collaboratively developed research questions, study designs, resources and data needed, 
measures to track collaboration and study progress, and the ultimate goal of the RPP. Also 


consider anticipated challenges, questions that remain, and next steps. 


As you develop the logic model, discuss how collaboration between members of the RPP 
enhances the collaboratively developed line of inquiry and informs continuous quality 
improvement efforts within the program and the research itself. While developing the logic model, 
imagine that you will be asked to present your ideas to a group of stakeholders that are key to the 
success of the RPP but unfamiliar with RPPs. Discuss the rationale for this RPP and how you might 


persuade the appropriate stakeholders to join and support it. 


As you develop the logic model and the rationale for the RPP, consider the following 


background questions: 


1. What do leaders, researchers, and practitioners need to understand about each other's 
culture and practice to collaborate effectively, understand program implementation, and 


examine program effectiveness? 


e How are the partners similar and distinct? 
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. What additional research questions remain about the implementation and effectiveness of 


the BPS early childhood program? 


¢ What additional data do you think could be collected in an RPP to answer 


those questions? 


. What should early-career researchers know and be able to do to conduct solid ECE 


implementation research in collaboration with policymakers and practitioners? Think not 
only in terms of content knowledge, but also in terms of ability to work with others and the 


personality traits that may or may not make that successful. 


4. What points would you make to policymakers and practitioners for how research findings 


could be applied in practice? 


Afterward, share and compare the various RPP logic models across the small groups. How are they 


similar or unique? Were there common themes? 


> 3. Addressing the need for tools. As Sachs notes in Chapter 7: The lion’s share of our PD focuses on first 


setting the table—getting teachers to understand their curriculum and the “whys” underneath it, and then getting them to 


reflect about whom they are teaching and how to differentiate their instruction. Though we focus on curriculum fidelity, 


we view it as “a tool, not a rule.” We know that strong teachers will need to make adjustments along the way to meet 


the diverse needs of their classrooms. The rub is getting them to make choices based on what facilitates learning versus 


what is easier fo manage (p. 159). 
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Researchers and policymakers need to help define student outcomes and what teachers should do. We 
need tools with specificity that take a fine-grain look at what happens between teachers and kids. To 
look at what works for which cultures, which languages, which income level and tools that let us see 

how income and culture interact. We don’t have these measures and we need them urgently... The best 
thing we did was to create a curriculum and fidelity tool and then we could celebrate and acknowledge 
what good instruction looks like. (Sachs interview, 11/17/2020) 
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We have tried a variety of coaching models, with ratios as low as one coach to eight teachers and as high as one 
coach per 20, which was more of a grade-level team focus. What we have learned is that coaching is most effective 
when the teacher wants to change and that the strategies we use need to be differentiated based on a teacher's 


knowledge level and how committed the school or program is to change (p. 158). 


We are interested in getting teachers to ask questions about big things such as, “What are necessary 
spaces?” so they can work together to talk about children’s work and to reflect on children’s thinking. 
(Sachs interview, 11/17/2020) 


Activity 


Fidelity tools are often used to assess how closely the delivery of a program aligns with the 
intended model and purpose when examining implementation. For this exercise, one aspect of a 
program or curricula has been identified to further your thinking about how to measure if program 
implementation matches the original program model and design. In small groups, choose a 
curriculum used by ECE programs in your local context and design a fidelity measure/tool to 
provide “a fine-grain look” at what happens between early educators and their students. Attempt 
to add the perspective of colleagues such as policymakers, practitioners, and parents. Consider 


the following questions: 


1. How can the needed tools that Sachs describes make a difference in supporting early 


educators in their work and supporting children’s outcomes? 
2. What research questions will the fidelity tool address? 


3. What dimensions of curriculum implementation and instructional practice will the fidelity 


tool measure? 
4. How will the data be analyzed to answer the research questions? 


5. In what context will this tool be used? Think about the ECE setting, geographic location, 
funding streams, racial/ethnic/linguistic breakdown of the children served, racial/ 


ethnic/linguistic breakdown of the workforce, etc. 
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6. What element of the practitioner-child interaction, instructional content, or children’s 
learning will be evaluated to assess fidelity? 
* How will they be studied? 
¢ What types of data will you collect and what measures will you use? 
7. How will the tool support instructional practice responsive to the various subgroups of 
children attending the program? 
8. How will you measure the reliability and validity of the tool? 


9. How much administrative demand and time will it take to be completed? 


¢ Who will be involved in documentation and observation? 
10. How will the tool be used by coaches to support instructional practice in classroom and 
home-based settings? 


11. How will the tool be used by practitioners to inform their own practice? 


12. In what ways will the information collected be valuable to stakeholders? Be specific about 


different types of stakeholders. 


* How will you use the information you collected from this measure to inform 
stakeholders? Describe different strategies you could use to disseminate this 


information. 


Afterward, share and compare the various fidelity tools across the small groups. How are they 


similar or unique? Were there common themes? 
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An Overview of Implementation Research and Frameworks in Early Care and Education 
Research introduces implementation science principles specific to researching the effectiveness 
of early care and education (ECE) programming. Authors JoAnn Hsueh, Tamara Halle, and 
Michelle Maier outline principles and frameworks from implementation science that undergird 
implementation research and point readers to additional volume chapters explaining how to 
use implementation research to improve the scaling of ECE programs across different settings 
and contexts. 


The summary above appears in the Getting it Right Chapter Summaries resource. 


With frameworks, we are trying to illuminate areas where research hasn’t focused. For example, 
how was a program designed? How is it being delivered? Is it reaching the intended population? 
In the literature, we often take a focused, myopic view of what makes a program effective, and 
we don’t pay attention to the broad constructs about how the program is being delivered and the 
contexts and circumstances that can also shape whether the program is effective. An implementation 
framework can help assure you have sufficient information to know what makes a program you are 
developing or scaling effective or not. (Hsueh interview, 10/13/20) 


The following themes, chapter excerpts from the Getting it Right full publication, and related questions are provided 


to stimulate additional engagement around the ideas and inquiry shared in this chapter. 


> 1. Defining implementation research. Implementation research encompasses the application of 
implementation science frameworks and principles to systematic inquiry into the act of carrying out a program, as well 
as a systematic inquiry into how a program is received and experienced in real-world settings and situations. In its 
most basic form, implementation research and analysis aim to illuminate what is happening, how it is happening, who 
is making it happen, why a program achieves the outcomes that it does, and for whom it works best. Implementation 
research can take a vertical perspective, looking at how processes across different levels of the supporting system 

can work in synergistic or countervailing ways to support a program’s implementation, or it can take a horizontal 
perspective, examining how implementation unfolds across a range of different settings, contexts, and populations 
(Ryan, Ch. 11; Vavrus & Bartlett, 2006) (p. 181). 
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Implementation frameworks serve as organizing tools that help highlight underexplored areas and point to ways to 
improve ECE program effectiveness for narrowing achievement gaps. By embedding the study of ECE programs within 
these frameworks, we can begin to broaden our knowledge of the influences that shape the lives and trajectories of 


young children, particularly those from low-income and racial, ethnic, and immigrant minority backgrounds (p. 189). 


As Hseuh explains, “There’s not a lot of familiarity about implementation frameworks in ECE. We want people to 
consider their research and operational implications as we try to illuminate areas where research hasn't focused. For 
example: How was a program or initiative designed? How is it being delivered? Is it reaching the intended population? 


For whom is the program impactful? 


This chapter may make it feel like you have to capture everything (in your research design), but the point of a 
framework is to plant seeds of where you can look. Where you can look critically at the literature about what evidence 


is known; where you can focus on measurement, strategies, and your design using elements of the framework.” 


Discussion Questions 


All programs are collecting information, but often it is limited to a small set of indicators. For 
example, how many kids attend? Do they attend regularly? Implementation research can point 
towards taking a more holistic approach. It is not just dosage of how many times someone attends, 
but it is about what do they get when they do come. What is their experience when they are there? 
How does that compare to a different program that may be out there, too? (Hseuh interview, 10/13/20) 


Research Design and Methods | Research Collaboration and Partnerships 


1. How can implementation frameworks serve as organizing tools to support researchers, 
policymakers, and practitioners in collaborating to identify a research question, develop a 


research design, collect data, and evaluate and communicate findings? 


2. Given the current context, what areas of inquiry are most critical for ECE implementation 


research to examine and for what purpose(s)? 


* Has this changed over time? 
¢ What research partners are needed? 


e¢ What information is needed? 
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3. What are the benefits and challenges in collaboration between researchers, policymakers, 


practitioners, and families to study how a program is delivered? 


* How might you address some of the challenges? 


> 2. Adopting an inward or outward focus on implementation. Implementation frameworks 

underscore where research can focus and, in turn, generate hypotheses and research questions. A growing set of 
implementation frameworks have been applied to ECE; one kind focuses inward on program components and 
structure, and another focuses outward on the contexts and larger infrastructure that support successful implementation 


of programs and systems... 


Given this interdependence, implementation researchers differ in their perspectives of what constitutes an inward or an 
outward focus. Indeed, these distinctions can shift with a researcher's focus of inquiry. For the purposes of this chapter, 
implementation research that focuses inward addresses a program’s theory of change or implementation processes, 
while implementation research that focuses outward is oriented to the larger context and infrastructure supports that 
surround a program. These foci highlight potential sources of variation that may account for the effectiveness (or lack 
thereof) of ECE programs, as well as for how such programs may have varying effects in different contexts and for 


children with different backgrounds (p. 182). 


When we want to influence policy and practice, looking outside the program to the larger context 
and infrastructure of supports around a program is key to answering the questions: Are beneficial 
services being delivered that children (and families) aren’t getting elsewhere? What is the value- 
add of this particular program? Many folks say, “Here’s the program we designed, and here’s its 
impact.” But without taking into consideration the surrounding context and critical supports and 
infrastructure that were in place to create the effective program, the field has very little information 
to guide replication of its effects or scaling in other contexts and situations. (Hseuh interview, 10/13/20) 


80 GETTING IT RIGHT: THE CONVERSATION GUIDE FOR PREPARING THE NEXT GENERATION OF IMPLEMENTATION RESEARCHERS FOUNDATION FOR CHILD DEVELOPMENT 


GETTING IT RIGHT GUIDE CHAPTER 8: AN OVERVIEW OF IMPLEMENTATION RESEARCH AND FRAMEWORKS IN EARLY CARE AND EDUCATION RESEARCH 


Discussion Questions 


Research Design and Methods | Program Evaluation | Research Approach and Stance 


1. Choose a particular research question that is of interest to you. How might an implementation 
framework and implementation principles influence the way you design a research study 


around this question? 


2. What are examples of research questions that might require a more inward focus on 


implementation? 


3. What are examples of research questions that might require a more outward focus on 


implementation? 
4. Why is it important to differentiate between an inward vs. outward focus on implementation? 


¢ How is implementation fidelity important in bothe 
p y imp 


¢ What are some examples of inter-relations between inward and outward focuses that 


should be considered in research designs? 


5. As researchers, how do you learn and gather data about services and opportunities offered to 


children and families in the community outside an ECE program? 
¢ What approach would you take to build research partnerships with these programs 
and program administrators? 
6. Why are these relationships important? 
7. What are the benefits of including community stakeholders and family members on your 
research team? 
¢ What are the challenges? 
¢ What are strategies that you could use to engage family members? 


8. Based on your experience, what are internal and external conditions that are more or less 
influential on children’s outcomes in ECE programs and can affect program quality across 


contexts and at scale? 


¢ How can implementation research help capture such conditions? 
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> 3. Focusing on stages of the implementation process. The National Implementation Research 
Network, for instance, identifies four implementation stages: exploration, installation, initial implementation, and full 


implementation (Bertram, Blase, & Fixsen, 2015') (p. 186). 


During exploration, stakeholders are assessing their needs and identifying what will best fit those needs in terms of 
adopting new programs, policies, or practices. They are also examining the feasibility of taking on a new practice, 
program, or policy, including assessing buy-in by all those affected by such a decision. During installation, the new 
program is not yet being delivered, but stakeholders are busy making sure that they have the technical, financial, and 
human resources fo carry it out. This may involve hiring and training new staff or training existing staff (i.e., addressing 
staff competencies) or making structural and instrumental changes organizationally (i.e., addressing organizational 
infrastructure) that enable stakeholders to carry out the new program. Initial implementation signals the start of service 
delivery. During this stage, data are regularly gathered and used to assess how well things are going and to adjust as 
necessary, with the goal of continuously improving implementation. Rapid-cycle problem solving becomes prominent 
during this stage and continues even when full implementation is achieved. Full implementation is characterized by 
skillful implementation of the new program, with the necessary skilled practitioners, organizational infrastructure, and 
leadership in place to support its continued reliable use and sustainability.2 While these stages are presented here in a 
sequential, linear order, there is consensus in the field of implementation science that the stages are recursive (Saldana, 
2014?), and that achieving full implementation of a well-defined evidence-based program can take between two and 
four years (Bierman et al., 2002; Fixsen, Blase, Timbers, & Wolf, 2001°) (p. 186-187). 


Often because research happens at a point in time, we assume the program, and ensuing 
findings, are static. But the program and the context supporting the program are always changing. 
For example, the population, economy, staff, and program evolve over time. As a program develops 
services, its theory of change may get more defined over time. If the program shows some success, 
it may gain new funding and serve new children, expand, and have satellite locations where it can 

reach more children. Every time a program expands, it comes into a new context. New staff may 
be hired. There is a dynamic interplay as both context and the program itself change. It requires 
a shift from seeing research as “a snapshot” to seeing it as “a process over time.” When data 
collection is embedded into a program’s daily life, there will always be a source of information 
about these dimensions. (Hsueh interview, 10/13/20) 


82 GETTING IT RIGHT: THE CONVERSATION GUIDE FOR PREPARING THE NEXT GENERATION OF IMPLEMENTATION RESEARCHERS FOUNDATION FOR CHILD DEVELOPMENT 


GETTING IT RIGHT GUIDE CHAPTER 8: AN OVERVIEW OF IMPLEMENTATION RESEARCH AND FRAMEWORKS IN EARLY CARE AND EDUCATION RESEARCH 


Discussion Questions 


Research Design and Methods | Program Evaluation 


1. Consider the ways that programs, and instructional practices within programs, may differ 
across the different stages of the implementation process. How might such differences influence 


your research design? 


How might this change the way you examine program and child outcomes during 


each stage? 


How might the ways you collect early educator or child-centered data change 


throughout the stages? 


Do the stages have any effect on how you collect classroom and home quality data? 


How might the ways you interpret the data change across the different stages of 


implementation? 


How might your overall study design change across the different stages of 


implementation? 


2. Pretend that you have set out to design a successful early childhood program. How might 
your program change as it develops through the different stages of program development and 


implementation? 
Pp 
* Do the question responses vary by professional role (e.g., director, early educator, 
coach, researcher) or by ECE setting? 


¢ What operational challenges might each professional role see at the various 


implementation stages? 


¢ What might be the advantages of blending research from both inward and outward 
perspectives while developing and assessing a program along different stages of 


implementation and program development? 
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> Moving forward. 


Any advancement you can make advances the field as a whole. It is the findings of many studies that 
will move the field forward by looking outside traditional research. People think, “my dissertation has 
to be the be-all, end-all.” Move in the direction of drawing upon implementation research paradigms 
and principles but focus on a subset. Balance the practical and the grander vision in your research. 
You are part of a movement that is bigger than yourself. You have something to contribute. (Hsueh 
interview, 10/13/20) 


Discussion Questions 


Research Approach and Stance 


1. How do Hsueh’s insights inform your perspective as a researcher today? 

2. What is Hsueh referring to when she says, “outside traditional research”? 

3. How might traditional and nontraditional research work together to move the field forward? 
4. What is an implementation question within the ECE field that you would like to study? 


* How will you hone this question in a way that contributes to moving the 
field forward? 
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In Designing Implementation Research to Guide the Scale-Up of Effective Early Care and 
Education Across Settings, researchers Michelle Maier and JoAnn Hsueh call for concerted 
efforts to design and enhance implementation research to better understand variation in 
implementation and program impacts from multiple and holistic perspectives. Expanding 

the scope of early care and education (ECE) to scale implementation research helps to ensure 
findings can be used to guide policy and practice as well as determine how best to support and 
sustain effective programming to reach a broad number of children and close disparities in 
achievement. 


The summary above appears in the Getting it Right Chapter Summaries resource. 


Our chapter is a call to researchers and practitioners to become more intentional in asking and 
answering questions, including: What are the goals? What has been put in place to meet those goals? 
What happens? What might be getting in the way? Answers to these questions are needed to inform 
adaptations or improvements that will help determine and strengthen outcomes. (Maier interview, 11/6/20) 


The following themes, chapter excerpts from the Getting it Right full publication, and related questions are provided 


to stimulate additional engagement around the ideas and inquiry shared in this chapter. 


> 1. Well-designed implementation research is the key link between small-scale early care and 
early childhood education (ECE) programs that have been proven to work and large-scale 
adaptations across populations and settings. Waiting years to see whether programs work provides too 
little information too late. Ongoing, well-designed implementation research, however, can provide real-time feedback 
on necessary program adjustments, identify the supports needed to successfully put these programs into action in 
varied localities and contexts (Martinez-Beck, 2016°), and address why and how a program works and under what 
circumstances. Such research gives the field the information it needs to bring promising programs to wider populations, 


enabling all children to have access to high-quality learning experiences (Phillips et al., 20177) (p. 197). 
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The goal is to try to implement with fidelity. Folks in the field question whether how programs unfold 
naturally is sufficient to generate positive impacts but may not be taking an active stance to collect 
ongoing data to empirically investigate and understand how things are going. We need to consider ways 
to formalize data collection in doable ways. (Maier interview, 11/6/20) 


What is tough is that there often isn’t a strong theory of change, or core components are not clearly 
articulated by developers of a program. Pulling that apart is critical to setting the foundation of where 
to start in collecting data. Implementation research is a call to take a step back—to think about what 
you are doing and why. Taking the time to do this is going to shape the program, the research, and 
determine and enhance outcomes. (Maier interview, 11/6/20) 


Discussion Questions 


Research Design and Methods | Research Collaboration and Partnerships 
1. How would you explain the potential benefits of using an implementation framework to your 
research partners (e.g., policymakers, practitioners)? 
* How would you explain the challenges? 
2. How can the context of an ECE program or intervention change during various stages of 


program implementation and scale-up? 


3. How can various contextual factors during implementation influence the attainment of the 


planned program and child outcomes? 


4. How can contextual information from implementation research help the various research 


partners replicate programs or specific practices? 


5. How would you manage expectations of research partners (e.g., policymakers 


and practitioners)? 
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> 2. Implementation frameworks guide the focus of research. This framework highlights where sources 


of variation may be likely to influence program effects and, therefore, underscores where research can focus. This 


includes operationalizing and measuring (p. 199): 
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* fidelity of implementation of the program and implementation plan; 

* proximal sources of variation in program effects such as treatment contrast, participant 
characteristics, and program context; 

* distal sources of variation such as characteristics of the implementing organization and of the larger 
system; and 


* potential moderators of these relationships. 


No one study needs to or can capture all of the elements in this framework. The point of this framework 
is to plant seeds of areas to study, to illuminate areas where research hasn’t focused. It is the findings of 
many implementation studies that will move the field forward. (Hsueh interview, 10/13/20) 


Discussion Questions 


Research Design and Methods | Program Evaluation | Research Collaboration and 


Partnerships 
1. What are examples of research questions that the conceptual framework depicted in Chapter 
8 that may help answer what works or doesn’t work, for whom, and under what conditions? 
¢ What questions could help answer how effects might vary across 
different contexts and populations? 


¢ What questions could help identify the benefits and challenges of 


maintaining fidelity to program models in application? 


¢ What questions could help answer how the “program planned” might 


be similar to or very different from the “program received”? 


2. How is an implementation framework different from other research methodologies 


and frameworks@ 
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3. Based on your experiences and knowledge of ECE, what sources of variation are more or less 


influential on programs and child outcomes at various stages of implementation? 


* How would this influence your research design? 


4. How could this implementation framework be useful to you when you work with policymakers 


and practitioners to define research questions and determine research design? 


> 3. Creating an evidence-building cycle. Embedded in each of these stages of program development are 
three aspects of evidence-building research (Knox, Hill, & Berlin, 2018; Metz et al., 2016): 


* implementation of the program model, which is continually in flux and evolving at each stage of 
program development; 

* adaptation of and adjustment and improvement to the defined program model, organizational and 
system supports, and infrastructure; and 


* building impact evidence by testing the program model. 


In essence, these evidence-building activities have a cyclical relationship; iterative feedback loops aim to strengthen the 
model as the circumstances, context, and environment in which the program is being delivered evolve, which in turn can 


help the program operate successfully at each new stage of program development (Knox et al., 2018). 


ECE can benefit by aligning implementation research designs and measurement to this evidence-building cycle 

and stages of program development. As Manno and Miller Gaubert (2016°) argue, (a) many implementation 
research topics and questions are relevant across stages, but depending on whether a program is undertaking 
horizontal or vertical scale-up, the specific research questions and their emphasis will be slightly different; and (b) 
even in early stages of program development, implementation research can lay important groundwork for informing 


future scale-up (p. 200). 
The authors illuminate potential areas of study within implementation research, including:? 


* Treatment planned, offered, and received 

¢ Implementation plan and system supports 

* Characteristics of participants 

* Characteristics of organizations implementing the program 

* Institutional and contextual factors external to an organization 


* Strength of service contrast resulting from the program 
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Being intentional—knowing what you are putting into place, what the goals are behind that, are 
you reaching your goals, and what might be the way of doing that—can help inform adaptations and 
improvements to achieve our goals: strong outcomes for all children. (Maier interview, 11/6/20) 


Discussion Questions 


Research Design and Methods | Program Evaluation | Research Approach and Stance 
1. What are the most common institutional or contextual differences that may arise in program 
implementation that could affect program and instructional quality and/or child outcomes? 
* Choose one of these factors and design a research study investigating 
the impact of their differences. 


¢ Why does investigating these contextual factors matter for your work 


and for other partners? 


¢ How might such investigations identify deeper questions about the root 


causes of inequity and ways to eliminate disparities? 
2. What kinds of information do researchers and their partners need at each stage of 


implementation? 


3. In what ways may implementation research questions, designs, and measures change at the 


different stages of implementation? 


A. Consider the three aspects of evidence-building research listed above and in the chapter. What 
difficulties might program administration and staff face during each of these stages, and how 


might you respond to those difficulties as a researcher? 


5. As a researcher, how might you decide when it is appropriate to examine child outcomes 
across the different stages of program implementation and development? 
¢ What factors might you need to consider? 
¢ What challenges might you face? 


6. Looking at the inward and outward focuses for implementation research in realife contexts, is 


there an end to an evidence-building cycle? Why or why not? 
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> 4. Potential methodological approaches to implementation research. Implementation studies can 
take multiple forms, using quantitative, qualitative, or mixed-method approaches. Quantitative efforts are more objective, 
closed-ended, and numerical in nature; use statistical analysis; and commonly rely on methods like surveys, direct 
assessments, structured observations, and administrative data. Qualitative efforts are more exploratory, subjective, and 
open-ended in nature and typically rely on one-on-one interviews or focus groups (conducted at a single time point or 
multiple time points), ethnographies, document reviews, unstructured or semi-structured observation, and case studies, 


among others. Mixed-method approaches combine these two types of methods (p. 201). 


We had tested Building Blocks, a math curriculum, and then folks we trained to be coaches for this 
evaluation moved over to the New York City Department of Education to put this curriculum in place. 
How to track implementation? Coaches filled out a log. At one point a coach said to me, “You forced me 
to learn about and do this log. | never understood why. | now understand that if | have 80 coaches doing 
this, there needs to be a mechanism to track fidelity.” This is an example of real research in the real 
world, (Maier interview, 11/6/20) 


Barriers in implementation research are often small and prevent human beings from doing things. For 
example, boxes in a journal may be too little to write in or not having a printer to print out materials. 
Researchers have to learn what will work from people on the ground. (Maier interview, 11/6/20) 


Discussion Questions 


Research Collaboration and Partnerships 


1. How might a collaboration between researchers, policymakers, and practitioners increase the 


possibility that varied methodological approaches are used? 


e How might you foster this collaboration? 
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Research Design and Methods | Program Evaluation 


1. What contributions can each research approach (e.g., quantitative, qualitative, and mixed 
method) make in understanding how, why, and for whom a policy, program, or practice does 


or does not work? 


¢ What are the advantages and disadvantages of each approach? 

¢ What different skills might a researcher need to utilize each approach? 

¢ Think about different research questions you might ask that would 
require a qualitative approach, a quantitative approach, and a mixed- 


method approach. 


2. How might mixed-method designs deepen understanding of positive child outcomes in 
small-scale model programs that can, in turn, be applied to large-scale adaptations across 


populations and settings? 


3. Maier lists writing space and printing issues as “small barriers” to carrying out implementation 


research in the real world. What other small barriers might you imagine? 


¢ What steps can be taken to prevent, remove, or mitigate such barriers, 


especially given the limited funding that research typically has? 


4. Choose one of the six topics of inquiry listed in Chapter 9. Design a research study around one 
of these topics. Consider the methodological approaches that you might use and why, as well 
as the measurement considerations you will have to take into account. Additionally, consider 


the small barriers that might exist and how you might work around those barriers. 


¢ Why did you choose this topic? 
¢ What impact could your study have? 
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> Moving forward. 


Maier says that practitioners and researchers often get so busy that they don’t take or have the time to pause and think 


about what they are doing and why, but doing so is key to getting it right. 


Discussion Questions 


Research Approach and Stance 


1. How can taking time to think about your approach enhance your effectiveness in learning 


about what works? 


2. How might taking the time to think about an approach enhance a policymaker’s or 


practitioner's effectiveness? 
¢ Why might policymakers and practitioners not have the time to do this? 
¢ What can be done to mitigate those barriers? 
3. In a very practical and concrete sense, what steps can you take to ensure that you are taking 


the time to think about what you are doing and why? 


4. What practical steps can be taken so that your research team can have the opportunity to think 


and talk in depth with colleagues on an ongoing basis? 
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In How Implementation Science and Improvement Science Can Work Together to Improve 
Early Care and Education, Tamara G. Halle explains the value of two frameworks to advance 
effective implementation and quality improvement in early childhood programs, policies, 
and practices. Despite nuanced differences between these approaches, they share enough 
similarities that they can be easily combined to support and promote evidence-based early 
childhood programs and systems by identifying what works in different contexts and 
conditions while providing insights for continuous improvement. 


The summary above appears in the Getting it Right Chapter Summaries resource. 


The bottom line is that what these frameworks are doing is organizing and helping stakeholders— 
including researchers, practitioners, and policymakers—to articulate and consider together what 
works, for whom, and under what conditions. Together they can help us see what we know and what 
we still need to know in order to effectively implement and improve services to children and families. 
(Halle interview, 11/13/20) 


The following themes, chapter excerpts from the Getting it Right full publication, and related questions are provided 


to stimulate additional engagement around the ideas and inquiry shared in this chapter. 


> 1. Implementation science is the systematic inquiry into the processes by which interventions 
are enacted in the real world. It examines not only the interventions themselves but also the contextual factors 
and organizational supports that are necessary to create a hospitable environment for enacted interventions to achieve 
their intended outcomes (Century & Cassata, 2016; Damschroder et al., 2009; Granger, Pokorny, & Taft, 2016; 
Martinez-Beck, 2013; Peters et al., 2013; Peters, Tran, & Adam, 2013). It typically focuses on the implementation of 

an evidence-based program or practice. Consequently, implementation science, like some program evaluations, is 
interested in intervention fidelity, that is, the extent to which the intervention was actually delivered “as designed” and 
intended (Hulleman, Rimm-Kaufman, & Abry, 2013) (p. 227). 


Improvement science involves a systematic examination of the methods and contextual factors that best facilitate 
quality improvement at the individual, program, and/or system level (Health Foundation, 2011; Langley et al., 
2009; Shojania & Grimshaw, 2005). Improvement science draws heavily on process improvement models from 
business and manufacturing (Deming, 1986) and on organizational change management theory (Cameron & Green, 
2009), as well as implementation science (Durlak & DuPre, 2008; Fixsen, Naoom, Blase, Friedman & Wallace, 2005; 


Meyers, Durlak, & Wandersman, 2012). Improvement science originated in manufacturing as the systematic study of 
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the series of steps and activities that make up a work process, with the aim of improving the quantity and/or quality 
of the work product and reducing costs. The inclusion of systems thinking and change management perspectives led 
to the study of how workers think together about improving their activities as a team. Improvement science strongly 


emphasizes the expertise of practitioners and their role as “active inquirers” who develop practice-based evidence 


(Bryk, 2015) (p. 227-228). 


What, then, distinguishes these frameworks? The distinctions are subtle. Implementation science tends to focus on the 
conditions that support fidelity to evidence-based or evidence-informed practices as a means to achieve the intended 
outcomes of an intervention, whereas improvement science does not (see Table 1). Rather, improvement science tends 
to focus on innovation and adaptation based on evidence-based practices as a means to achieve improved outcomes. 
However, implementation science also acknowledges and tests adaptations and is interested in improved outcomes, not 
just fidelity and intended outcomes (Century & Cassata, 2016) (p. 229). See Table 1. 


These frameworks came from very different disciplines—business and manufacturing and health care. 
They are helpful and can help us improve services to children and families. (Halle interview, 11/13/20) 


Discussion Questions 


Research Design and Methods | Program Evaluation 


1. Why are implementation science and improvement science being brought to bear at this time 
to support the work of the early childhood education (ECE) field to achieve outcomes for 


children, especially those from low-income backgrounds? 


¢ What perspective does each approach bring? 
¢ What are the implications of an emphasis on innovation and adaptation of 


practice to fit the current context rather than fidelity to set program standards? 
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2. In what ways can implementation science and improvement science approaches contribute to 
moving the ECE field forward? 


* How do they do so separately? 

* How might they do so together? 

¢ What are the implications for research, policy, and practice in recognizing 
that evidence-based practices may need to be adapted to work in different 
contexts or for different individuals in new settings? 

¢ What is the importance of focusing on the use of data, implementation teams, 


and infrastructure throughout all stages of implementation? 


3. Identify a research question about program evaluation and effectiveness and design a study 


using either implementation science or improvement science to answer it. 


¢ What considerations led you to that particular approach? 


* How will your decisions support your work better than the alternative? 


> 2. Conceptual model incorporating implementation elements into traditional program and 
policy evaluations. Figure 1. (p. 233) 


: ete Program/Policy Program/Policy _ 
A —P Priority/Program —> Activities —> Outputs —> Implementation ShortT Long-Term 
Selection Outcomes —- Beret Program/Policy 
Program/Policy Ones 
1 + T t T Acceptability, Adoption, Outcomes 
Appropriateness, Deenieatanl Organizational, 
Leadership Readiness | i a esey ne Servite Proctor, Sarvs 
& Stakeholder Implementation Cost, Coverage, pag cA & Client/Recipient 
> Supports ————— il Sustainabil & Client/Recipient 
Engagement Individual, ustainability anne Outcomes 
t CUE Implementation Teams ulti 
System Implementation 
Infrastructure, Data 
Goals & & Feedhack Loops 


Feedback 


CONTEXTUAL FACTORS 


Internal Context (e.g. organizational climate) External Context (¢.g. policy environment) 


Note: Incorporates concepts from Bauer et al. (2015), Brennan et al. (2013), Damschroder et al. (2009), Metz et al. (2015), and Proctor et al. (2011). 
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Figure 1. It says to researchers: “Paying attention to and finding measures that let you look 

at supports are important factors to capture in your evaluation. What the field needs is better measures 
of implementation supports. We need more validated measures in particular; but for now, we need 

to collect information about implementation supports qualitatively if not quantitatively in our program 
and policy evaluations. This will help us tell the story of what works, for whom, and under what 
conditions. And we need to write about these measures and findings in our research reports and 
journal articles. The more we share this in our reports, the more we will be able to build the field.” 
(Halle interview, 11/13/20) 


Figure 1. It says to practitioners: “Everyone is a stakeholder, everyone a leader, everyone a 
researcher. That is the ideal. To create a hospitable environment to sustain continuous quality 
improvement and to sustain best practices with the goal of achieving the desired, improved outcomes. 
This has to do with attitudes and mindset and with programmatic support and organizational 
structures. Examples of such programmatic support include time for staff to meet and talk about what 
is working for them; using staff meetings to talk about what works; and modeling data use in decision- 


making to test whether something is leading to improvements in practice.” (Halle interview, 11/13/20) 


Discussion Questions 


Research Design and Methods | Program Evaluation | Research Approach and Stance 


1. How does knowing about the resources and supports in place in program implementation 
enhance a researcher's examination and understanding of what works, for whom, and under 


what conditions2 


2. What types of data can be collected about implementation supports qualitatively and 


quantitatively? 


3. What do you see as the impact of emphasizing the expertise of policymakers and their role 
as “decision makers” in creating systems and allocating resources that support programs and 


practice examined in your research? 


4. What do you see as the impact of emphasizing the expertise of practitioners and their roles as 


“leaders” and “active inquirers” who develop practice-based evidence in your research? 


5. What are the benefits of a feedback loop for research stakeholders? 
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6. How can feedback loops help communicate research findings and develop practical 


applications? 


7. Consider the comparison of research questions and outcomes of interest in Table 2 (p. 234) 
and choose one that falls under implementation science and improvement science. How does 


each framework change the way you consider the research question/outcome? 


* How does each framework change the way you might design a study 
around your chosen question/outcome? What other differences exist 


between the two frameworks? 


> 3. Research and evaluation design: home visiting as a real-world example. 


Implementation science and improvement science argue for more practical and nimbler program development 
and for evaluation designs that can uncover the critical ingredients leading to successful implementation of early 
childhood interventions. Though some of these research design elements can be embedded in randomized control 
trials (RCTs), other innovative evaluation designs allow researchers, policymakers, and program designers to test 
innovations, identify important variability (Bryk, 2015), and get relatively quick answers to questions about what works 


for whom under what circumstances (p. 235). 


As with most evaluations and continuous improvement efforts, asking the right questions and getting them 


answered produces better outcomes (p. 231). 


Questions about data and feedback loops are related to another unique contribution of implementation science to 
program evaluation: the assessment of the existence, functioning, and quality of the implementation infrastructure to 


support an early childhood intervention model (p. 232). 


Home visiting models have been the subject of many traditional program evaluations over the years. For example, 
the Home Visiting Evidence of Effectiveness (HomVEE) project, supported by the U.S. Department of Health and Human 
Services, recently reviewed the research evidence for 20 home visiting models (Sama-Miller et al., 2018). HomVEE 
includes evidence of effectiveness from well-designed, well-executed RCTs and quasi-experimental designs. Most 
evaluations of home visiting models measure participant outcomes targeted by the interventions, such as parenting 
practices, family functioning, child health and development, maternal health and mental health, child abuse and 
neglect, or maternal life course outcomes such as deferral of subsequent births (Gomby, Culross, & Behrman, 1999; 
Sama-Miller et al., 2018). As models have matured, longer-term outcomes have been monitored, such as reductions in 


juvenile delinquency, family violence, crime, and family economic self-sufficiency (Sama-Miller et al., 2018) (p. 240). 
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The home visiting field has also embraced a focus on continuous quality improvement. In 2013, the Home Visiting 
Collaborative Improvement and Innovation Network (HV CollN) was established by the Health Resources and 
Services Administration (HRSA) to accelerate improvement among MIECHV grantees (p. 245). 


The HV CollN was active from September 2013 through August 2017. It demonstrated improvements in home visitors’ 
knowledge and skills in the topical areas, as well as an increase in the use of data to achieve improvements in the 
targeted outcomes. However, it did not achieve the ambitious levels of performance hoped for across all performance 
metrics. For example, the rates of exclusive breastfeeding at 3 and 6 months rose only 3% instead of the hoped-for 
20%. Specifically, exclusive breastfeeding at 3 months rose from 10% at baseline to 13.5% at the end of the CollN, 
and exclusive breastfeeding at 6 months rose from 5% at baseline to 8% at the end of the CollIN (Arbour, Mackrain, 
Fitzgerald, & Atwood, 2018). 


Nevertheless, the HV ColIN was deemed successful in demonstrating that home visiting outcomes could be improved 
through this QI method, and many tools and resources were created through the HV CollN that could help spread and 
scale up improvement efforts among MIECHV grantees, potentially even those that had not participated in the CollN. 
As a result, a second, 4-year HV CollN (called HV ColIN 2.0) was initiated in September 2017. HV CollN 2.0 will 
engage 25 state and territory MIECHV awardees and 250 local home visiting agencies in quality improvement efforts 
around two topic areas that were addressed in the first ColIN: (a) maternal depression screening, access to treatment, 
and symptom reduction, and (b) early detection of and linkage to services for developmental risk. In addition, the 
collaborative teams in HV CollN 2.0 will develop, test, and spread improvements in three new topical areas, the first 
of which is intimate partner violence.”° Awardees will be selected in three waves. Each wave will last about 12 to 18 


months and will once again use the BSC [Breakthrough Series Collaborative] framework for quality improvement. 


In sum, although improvements in performance metrics have been modest, positive qualitative outcomes associated 
with improvement science frameworks have led to additional investments in home visiting quality improvement 
collaboratives. Methods that focus on changing organizational climate to support continuous improvement seem 
promising compared to other quality improvement approaches that take a more individualized approach, such as one- 
on-one coaching. Early childhood researchers await with much interest and anticipation further evidence on the spread 
and sustainability of QI methods within organizations that participate in a BSC or CollN, as well as achievement of 


target performance metrics for the content addressed by these quality improvement models (p. 248-249). 


Early childhood professionals should feel empowered to think of themselves as researchers and to collect 
data to see if they are reaching their goals. Ideally, they should be intentional about making a change and 
gathering information about if that change resulted in improvement. This is how feedback loops should 
look—collecting data and feeding it back into practice and systems change. (Halle interview, 11/13/20) 
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Discussion Questions 


Research Design and Methods | Program Evaluation 


1. Discuss the tension between measuring fidelity to a model and documenting adaptation or 


customization. What considerations are part of this conversation? 


¢ What difficulties might arise in measuring fidelity? 


¢ What difficulties might arise in documenting adaptation or customization? 


2. What steps can be taken to help policymakers and practitioners gain access to 
research findings sooner? 
¢ What are the benefits of this? 


* How can findings be best integrated into instructional practice? 


3. What do the studies on home visiting reveal about implementation context and program 


implementation? 


4. What research questions regarding the implementation of home visiting programs lend 


themselves to an implementation vs. improvement science approach? 


> Moving forward. 


For the next series of questions, think about the case study home-visiting example (p. 242 and p. 245). 


Discussion Questions 


Research Design and Methods | Program Evaluation 


1. What are the lessons learned for the ECE field about the importance of implementation 


infrastructure and supports to influence practice? 


¢ What questions remain? 


2. How did the implementation studies support continuous quality improvement? 
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In The Contributions of Qualitative Research to Understanding Implementation of Early 
Childhood Policies and Programs, Sharon Ryan argues that qualitative studies examining 

the implementation of early childhood programs can provide practical information to help 
policymakers and leaders understand why early childhood programs do or do not fulfill 
their promise. The early childhood field has assumed that with evidence of best practices, it is 
possible to scale up what works in one site to many programs. Yet, evidence-based practices 
are often transformed, adapted, or even ignored in local sites. Therefore, it is imperative to 
look across programs at a macro scale while also employing qualitative studies to go deeply 
into variations in context and implementation strategies. With more qualitative studies of 
implementation across multiple sites, it might be possible to identify which local adaptations 
make sense and which may unnecessarily undermine best practices for young children. 


The summary above appears in the Getting it Right Chapter Summaries resource. 


This chapter tries to elevate the voice of practitioners through qualitative research. 

They are the ones who make a difference for children and families. We've been remiss as a field 
to not use mixed research method designs more often, even though we need both qualitative and 
quantitative research to answer basic questions about what works and what adaptations are needed 
if all young children are to benefit from best practice. These research approaches complement each 
other as long as they are rigorously done. (Ryan interview, 11/5/20) 


The following themes, chapter excerpts from the Getting it Right full publication, and related questions are provided 


to stimulate additional engagement around the ideas and inquiry shared in this chapter. 
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> 1. More recently, implementation researchers have begun to theorize about implementation 
as enactment—a complicated network of relations that assumes the movement from innovation 
to practice is multidirectional, not just top down or bottom up, as well as deeply political 
(Datnow, 2006; Honig, 2006). From this perspective, the implementation process is influenced and shaped by many 
agents (from children to policymakers) with varying levels of power and influence within educational settings that 
constitute a nexus of multiple policies at any one time. Researchers working from an enactment perspective look at 

the politics of innovation, and how a wide range of stakeholders working in various networks resist, transform, and 


implement policy depending on organizational ethos and resources, professional theories, and perceived need (Braun, 


Maguire, & Ball, 2010) (p. 262). 


Many people conducting research in our field don’t know what it is like to be in the classroom, 
to run a program, or to run a site. They are good researchers but have not had hands-on experience. 
Now there is a group of researchers trying to explain what it is like to be on the ground on the front 

lines and the challenge of implementing a program and policy program... We need to learn about 
the work with and from them. (Ryan interview, 11/3/20) 


Discussion Questions 


Research Design and Methods | Research Approach and Stance 


1. What does “implementation as enactment” suggest about the importance of researchers 
understanding the goals and daily life of the educators, children, families, leaders, classrooms, 


and programs being studied? 


¢ What are the implications of this perspective for the work of a researcher? 

¢ What does this suggest about the value of qualitative research? 

¢ What does this suggest about the development of research questions? 

¢ What does this suggest about collaboration between researchers and practitioners? 
¢ What does this suggest about collaboration between researchers and policymakers? 


¢ What does this suggest about collaboration between researchers and children? 
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2. What is your understanding of indicators of rigorous qualitative research? 


3. What are examples of how qualitative research can be used across the various stages of 


program implementation? 


4. How can researchers use qualitative and quantitative approaches to fill in gaps in their 
research and understanding of issues? 


5. How do qualitative and quantitative methodologies complement each other? 


> 2. Qualitative or interpretive research is interested in how individuals construct their social 
worlds and how those worlds are mediated by context and culture (Glesne & Peshkin, 1992). 
Research from this perspective typically involves spending a lot of time in educational settings, 
observing and talking with participants to develop an understanding and interpretation of 


educational phenomena. Qualitative researchers interested in implementation therefore examine innovations 


in sites of practice, often observing what takes place in schools and early childhood settings; they also shadow key 


stakeholders (leaders, teachers, families, state-level policymakers, coaches, etc.) and question them about an innovation 


and the reasoning behind their approach to implementing it. Using both the mutual adaptation and the enactment 


perspective, this research tends to focus mostly on the implementation of various public policies guiding prekindergarten 


or preschool (p. 263). 


106 


Qualitative research can provide portraits of practice so you can imagine what it is like to be in that 
classroom setting, community, or what it is like to be that kid. (Ryan interview, 11/3/20) 


Discussion Questions 


Research Approach and Stance 


1. What does qualitative research capture that is different or missed through quantitative research 
methods? 


2. What types of specific research questions could you answer with qualitative research that you 
cannot answer with quantitative research? 


3. What are the implications for the field that there are fewer qualitative research studies? How 


can the field better prepare researchers to do such work? 
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4. What would you say to a potential funder to advocate for additional qualitative studies? 


Research Design and Methods 


1. How can qualitative research help to understand: leadership, organizational climate, 
contextual factors (e.g., setting, location, resource allocation, participant experience, and 


perspective), policy, and practice in application? 


2. What other factors, apart from those listed in the text, might mediate the implementation of any 


educational reform? 


3. Why is it important for qualitative research to examine relationships (or lack thereof) between 


prekindergarten (including infant, toddler, and preschool) and K-12 systems? 


¢ What might be revealed about philosophical and instructional goals? 
¢ What might be learned about fade-out? 


¢ What might be learned about equity and ensuring strong outcomes for all children? 


4. How can qualitative research contribute to an agreed-upon vision and understanding of a 


continuum of education for children birth to grade 12 to move the field forward? 


¢ What would you want to communicate about philosophical and instruction goals? 
¢ What would you want to communicate about fade-out? 
¢ What would you want to communicate about equity and ensuring strong outcomes 


for all children? 


> 3. Toward a quality implementation research agenda. Focusing on the implementation of early 
childhood programming in local sites of practice and on the perspectives of participants helps us understand whether 
and to what extent a policy is implemented as intended, makes it possible to see how policies and programs are 
shaped by context and local actors, and can help with theorizing change and improvements in practice. However, the 
research base is limited to a handful of studies, and few of these look at implementation across multiple sites, multiple 
states, or at all levels of the system. The research reviewed in this paper suggests three possible paths toward a more 
comprehensive, critical, and policy-capturing use of qualitative research to improve the implementation of high-quality 
early childhood education systems. These include moving beyond classrooms and school districts to investigate multiple 
levels of the early childhood system, focusing on multiple stakeholders in the early childhood system, and, finally, 


considering equity (p. 269). 
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There are times when, to understand implementation, you should start with rigorous qualitative work. 
My work calls for consideration of qualitative research to understand equity for teachers and kids... 


You have to hang out every day to understand the subtleties—the decisions that are made every day. 
(Ryan interview, 11/2/20) 


You can gain a deeper understanding of issues like equity that can’t be captured in numbers. Qualitative 
and quantitative research can complement and enhance each other as long as they are rigorously done. 
(Ryan interview, 11/3/20) 


Discussion Questions 


Research Design and Methods | Research Approach and Stance 


1. Ryan suggests possible paths toward using qualitative research to improve the implementation 
of high-quality early childhood education (ECE) systems, such as investigating multiple levels of 
the system, focusing on all stakeholders, and examining issues of equity. What is similar about 
these paths? Different? 


* Choose one path you would focus on first and explain why. 


2. How can qualitative research capture social, economic, and political factors that might be 


influencing program or policy implementation? 


* Pick a research question that you are interested in. Think about what a qualitative 
study in two very different communities seeking to answer that question might 
look like. How would the social, economic, and political factors in each of those 
contexts influence your research? Think about your methods, your data collection 
procedures, and how you might interpret the data. 

¢ What factors might researchers consider as they decide whether to examine 
an innovation vertically, horizontally, or across stages of development using a 
qualitative approach? 


¢ What are the possible benefits and limitations of each of these approaches? 
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> Moving forward. 


“It is one thing to learn research methodology and another to think conceptually and be able to see 
where your work fits in the field... In our field, your approach depends on how you are mentored 
rather than taking a look at what the field needs. You are lucky if you study with a researcher who 
is flexible in using methods. You have to build your skill set and be open to different ideas, to taking 
some risks.” (Ryan interview, 11/2/20) 


Discussion Questions 


Research Approach and Stance 


1. Why is it so important that researchers understand the daily life and context of a program they 


intend to study? 


2. Why is it important for researchers to be flexible in what research methods can be utilized in 


implementation studies? 


3. How might your responses to the questions in Sections 1, 2, and 3 above intentionally or 


unintentionally shape you as a researcher? 
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SECTION 3, CHAPTER 12 


EQUITY AS A PERSPECTIVE FOR 
IMPLEMENTATION RESEARCH IN 
THE EARLY CHILDHOOD FIELD 


Milagros Nores, Ph.D., National Institute tor Early Education Research 
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In Equity as a Perspective for Implementation Research in the Early Childhood Field, Milagros 
Nores argues that addressing equity in implementation research is important to shape early 
childhood development investments and programs, particularly given that many of these have 
expanded under the principle of reducing inequities and disadvantages before kindergarten. 
Research with an equity lens helps define inequities in present conditions that may determine 
outcomes, ensures that the research itself does not introduce biases, and captures the extent to 
which programs and policies reduce or increase inequities. 


The summary above appears in the Getting it Right Chapter Summaries resource. 


Start with the research questions. Are you trying to understand differences? If you don’t think about 
this from the beginning, equity will not be addressed. Then think about your methods so that you 
can dig into differences in opportunities. If the goal is to learn about and improve equity, we have to 
reach out fo others to learn. To admit, “My ideas may be wrong.” To be flexible. To modify research 
questions and our approach as needed to keep the thread of equity running throughout the work. 


| do acknowledge that it is an ambitious goal, but at least we have equity in mind and we are trying 
to go in that direction... Ultimately, at the end of the road, we want to bring out voices of all sectors. 
(Nores interview, 10/21/20) 


The following themes, chapter excerpts from the Getting it Right full publication, and related questions are provided 


to stimulate additional engagement around the ideas and inquiry shared in this chapter. 


> 1. Research can not only help bring to light what works in the early years but can also 
document how programs contribute to increasing equity (or reducing inequity) and at what 
point in the education process they do so. That is, it can help us understand the effectiveness, efficiency, 
relevance, impact, and sustainability of ECED programs with respect to equity goals... Yet we cannot escape the fact 
that research itself—and the measures, researchers, observers, interviewers and other agents of research-may 
introduce biases of its own to any evaluation process. And if questions pertaining to equity are not asked, then equity 


is not assessed at all (p. 278). 
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Equity in research implies capturing the extent to which programs, policies, and interventions reduce or increase 
inequities, validly defining inequities in relation to the context and the disadvantages that are present, and integrating 
the concept of equity into all components of research, from the questions asked to the analysis and interpretation stage. 
In sum, understanding equity means being able to answer questions that attend to equity concerns. Who are the less 


advantaged, and how does this evaluation capture their experience with ECED policies and programs (p. 279)? 


Are your questions addressing equity in any way? If you don’t consider this at the beginning, equity 
won't be addressed... Be thoughtful about where to embed equity, trying to keep perspective 
alive throughout... Research has to be fluid when trying to engage individuals and families with 
experiences and obligations. Keep it alive—the equity thread has to be alive throughout. This is what 
allows you to be intentional—to make decisions with a purpose. (Nores interview, 10/21/20) 


Discussion Questions 


Research Design and Methods | Research Approach and Stance 


1. How can practitioners and researchers identify their own implicit and explicit biases? 
2. What strategies can be used to reduce or mitigate bias in research and practice? 


¢ What advantages could collaboration in research have to reduce or mitigate 


these biases? 


3. What effects might your own personal biases have on the research you conduct, interpret, 


and disseminate? 


4. What biases and inequities exist in current early care and education (ECE) systems, research, 
and practice? How do these biases and inequities impact the experiences and outcomes of 


children participating in ECE programs? 


5. Since researchers, observers, and interviewers may introduce their own biases in the evaluation 
process, how do you suggest researchers avoid exacerbating inequities across the various 


stages of the research process? 


6. What are the implications for the field and the children and families served if equity is not 


integrated into all components of research? 


7. What research practices lead to enhancing multicultural validity across the research process, 


and especially in measurement and assessment? 


FOUNDATION FOR CHILD DEVELOPMENT GETTING IT RIGHT: THE CONVERSATION GUIDE FOR PREPARING THE NEXT GENERATION OF IMPLEMENTATION RESEARCHERS 113 


GETTING IT RIGHT GUIDE CHAPTER 12: EQUITY AS A PERSPECTIVE FOR IMPLEMENTATION RESEARCH IN THE EARLY CHILDHOOD FIELD 


8. How can the preparation of ECE researchers better equip students to conduct research that 


does not exacerbate inequities? 


9. What knowledge and skills do you think you need to prevent your research from perpetuating 
or exacerbating inequities? 


10. How can research help support efforts to implement anti-bias, anti-racist instructional 


approaches? 


¢ What are the challenges in acknowledging and addressing these inequities? 


> 2. Equity, cultural competence and responsiveness, and intersectional approaches all 
interconnect in central ways in the design, collection, analyses, and interpretation stages 

of the research work. At their core is an emphasis on understanding the complexity of social and power 
dynamics and an explicit attempt to recognize, measure, and assess differences, as well as reduce biases (as much 
as possible) and employ culturally appropriate methods (p. 281). 


Maybe your questions address equity. Be open to modifying your questions and approach as 
you collect data. You start looking at data, why does it look in a certain pattern? Maybe there is 
something you missed... Maybe your instruments need to be adjusted depending on all that you 
learn. There are so many differences, including gender, disability, race/ethnicity, language, minority 
status, or religion... Which one are you trying to surface? They are not independent. Trying to surface 
some to make them visible can lead to actions that address them. (Nores interview, 10/21/20) 


Discussion Questions 


Research Design and Methods | Research Approach and Stance 


1. In what ways is equity-focused implementation research dependent upon what the researcher 
brings to the work as an individual? 


2. How is research enhanced when it is grounded in equity-based perspectives, cultural 


competence, and intersectional approaches? 
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3. How do you explain the differences between the terms “culturally competent” and 


a“ « uv 
culturally responsive”? 


¢ As a researcher, how could you be more culturally competent? 


* How could you be more culturally responsive? 


4. How can an equity perspective, cultural competence, and intersectional approaches be 
promoted in research involving populations from diverse racial, ethnic, cultural, linguistic, 
and socioeconomic backgrounds, especially when these factors are not independent of one 


another and may differ from yours? 


¢ What personal/professional traits are required to do so? 

¢ How can you promote these traits within yourself and within other researchers 
on your team? 

¢ What questions and thought processes could you suggest? 

* How does research practice need to change to promote the factors listed above? 

¢ How might interpretation, dissemination, and application of your findings need 
to be handled? 

¢ How would you consider the different audiences digesting and implementing 


your research? 


5. How can researchers actively engage all stakeholders across the research process to promote 


equity and cultural competence? 


> 3. Components of research. Thomas and McKie (2006) provide examples of how researchers’ values, beliefs, 
and biases can compromise an evaluation process. The questions asked and the questions not asked, what is focused 
on versus what is minimized, the evaluation approach selected versus the one discarded, the data collected versus 

the data disregarded, the interpretations made, and how and to whom the results are presented can all undermine an 


evaluation. 


An approach to research that truly incorporates equity requires integrating equity concepts across all these 


components, from questions asked to interpretation (Hood, Hopson, & Kirkhart, 2015) (p. 284). 
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Many of us are trained in one way of doing research. Reach out to others. Think of yourself as coming 
together and questioning together. If the goal is to improve and capture aspects of equity that will 
improve ECE for all, come with the view of growing yourself and your team. If I’m not the right 
person to do a study (e.g., Native Americans), that leads me to look to others. It still could be my 
research, my team, but | step back and someone representing the community may need to move to 
the front. (Nores interview, 10/21/20) 


Discussion Questions 


Research Design and Methods | Research Approach and Stance 


1. Choose three components of research from those listed in Chapter 12. In your role as a 


researcher: 


¢ What are the potential equity considerations for each? 

* Do they change by component? 

* Describe how you could change what you do to address inequities. 
¢ What are the challenges in doing so? 


* How do you reduce or mitigate bias? 


> Moving forward. 


“| try to be humble. | strive to have an equity lens. As a researcher, it is impossible to address it all. In 
some places, issues of equity are less obvious. For example, race in a program that is 90% white. 

You still have gender, special ed, language. In a more rich context-you can see them—they are more 
obvious... Issues of equity exist everywhere. I’m not expecting myself or any researcher to see or know 
it all... But to be open to reaching out to learn. There is a whole other world out there.” 

(Nores interview, 10/21/20) 
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Discussion Questions 


Research Approach and Stance 


1. As you look across your work using an equity lens, what do you see differently than you 


did before? 


¢ What do you see differently about yourself as a researcher? 

¢ What do you see differently about your team? 

¢ What do you see differently about your research questions, methods, data 
collection, analysis, and dissemination of findings? 

¢ What do you see differently about your communication and engagement with 
research partners and subjects of study? 

¢ How can you ensure that you continue to use this equity lens moving forward in 


your work and your career? 
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ADDITIONAL 
ez 


You may be at the point of collaborating with colleagues to integrate implementation research even more effectively 
into your work. Or perhaps you are introducing this valuable approach into your research for the first time. To support 


and guide you in moving forward, here are related resources: 


Barnett, W. S., Kwanghee, J., Youn, M. J., & Frede, E. (2013). Abbott Preschool Program longitudinal effects study: Fifth grade follow-up. Retrieved from 
National Institute for Early Education Research: http://nieer.org/research-report/201311 apples205th20grade-pdf 


Brooks-Gunn, J., Markman-Pithers, L., & Rouse, C. E. (Eds.). (2016). Starting early: Education from prekindergarten to third grade (special issue). The Future of 
Children, 26(2). 


Brooks-Gunn, J., McLanahan, S. (Eds.). (2005). School readiness: Closing racial and ethnic gaps. The Future of Children, 15(1). 
Charmaz, K. (2014). Constructing grounded theory (Introducing qualitative methods series) (2nd ed.). SAGE Publications, Inc. 


Coburn, C.E., & Penuel, W.R. (2016). Research-practice partnerships in education: Outcomes, dynamics, and open questions. Educational Researcher, 45(1) 


48-54. https://doi.org/10.3102/0013189X 16631750 


Early, D.M., lruka, |.U., Ritchie, S., Barbarin, O.A., Winn, D.-M. C., Crawford, G.M., ... & Pianta, R.C. (2010). How do pre-kindergarteners spend their time? 
Gender, ethnicity, and income as predictors of experiences in pre-kindergarten classrooms. Early Childhood Research Quarterly, 25(2), 177-193. doi: 
10.1016/j.ecresq.2009.10.003 


Espinosa, L. M. PreK-3rd: Challenging common myths about dual language learners, an update to the seminal 2008 report. Foundation for Child 


Development. https://www.fcd-us.org/assets/2016/04/Challenging-Common-Myths-Update.pdf 


Foundation for Child Development. (2018). The NYC Early Childhood Research Network: A model for integrating research, policy, and practice. Foundation 
for Child Development. https://www.fcd-us.org/sharing-knowledge-better-implementation-beginning-end 


Foundation for Child Development. (2020). Resilience and risk at the intersection of immigration and child well-being: Research insights from the Young 
Scholars Program. Foundation for Child Development. https://www.fcd-us.org/resilience-and-risk-at-the-intersection-of-immigration-and-child-well-being- 


research-insights-from-the-young-scholars-program/ 


Foundation for Child Development. (2020). Getting it Right: Using Implementation Research to Improve Outcomes in Early Care 


and Education. New York, NY: Foundation for Child Development. https://www.fcd-us.org/assets/2020/06/GettingitRight 
UsinglmplementationResearchtolmproveOutcomesinECE_2020.pdf 


Gullo, D.F., & Grave, M.E. (Eds.). (2020). Scientific influences on early childhood education: From diverse perspectives to common practices (1st ed.). 


Routledge. https://doi.org/10.4324/9780429468285 


Halle, T., Metz, A., & Martinez-Beck, |. (2013). Applying implementation science in early childhood programs and systems. Brooks Publishing. 


Halle, T. Paulsell, D., Daily, S., Douglass, A., Moodie, S., & Metz, A. (2015). Implementing parenting interventions in early care and education settings: 
A guidebook for implementation (OPRE 2015-94). Office of Planning, Research and Evaluation, Administration for Children and Families, U.S. Department of 
Health and Human Services. https://www.acf.hhs.gov/sites/default/files/opre/parenting_ implementation guidebook 109 b5082.pdf 


Hamilton, D., Darity, W., Price, A., Sridham, V., and Tippet, R. (2015). Umbrellas don’t make it rain: Why studying and working hard isn’t enough 
for black americans: The New School: Duke University: Insight Center for Community Economic Development. https://gallery.mailchimp.com/ 
bf2b9b3cf3fdd8861943fca2f/files/Umbrellas_Dont_Make_It Rain8.pdf 


Hannah-Jones, N. (2020). What is owed. The New York Times Magazine. 
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Henrick, E.C., Cobb, P., Penuel, W.R., Jackson, K., & Clark, T. (2017). Assessing research-practice partnerships: Five dimensions of effectiveness. William T. 
Grant Foundation. https://wtgrantfoundation.org/library/uploads/2017/10/Assessing-Research-Practice-Partnerships.pdf 


Institute of Medicine and National Research Council. (2015). Transforming the workforce for children birth through age 8: A unifying foundation. 
The National Academies Press. https://doi.org/10.17226/19401 


Institute of Medicine and National Research Council. (2000). From neurons to neighborhoods: The science of early childhood development. The National 
Academies Press. https://doi.org/10.17226/9824 


Iruka, |. (2019, September/October). Race and racism in early childhood education: Redressing the past to forge forward. Exchange Press. https://www. 
childcareexchange.com/article/race-and-racism-in-early-childhood-education-redressing-the-past-to-forge-forward/5024932 


Maxwell, J. A. (2013). Qualitative research design: An interactive approach (applied social research methods) (3rd ed.). SAGE Publications, Inc. 


National Academies of Sciences, Engineering, and Medicine. (2017). Promoting the educational success of children and youth learning English: Promising 


futures. The National Academies Press. httos://doi.org/10.17226/24677 


National Academies of Sciences, Engineering, and Medicine. (2018). Transforming the financing of early care and education. The National Academies 


Press. https://doi.org/10.17226/24984 


National Research Council. (2001). Eager to learn: Educating our preschoolers. The National Academies Press. https://doi.org/10.17226/9745 
National Research Council. (2008). Early childhood assessment: Why, what, and how. The National Academies Press. https://doi.org/10.17226/12446 


Phillips, D., Gormley, W. T., & Anderson, S. (2016). The effects of Tulsa’s CAP Head Start Program on middle-school academic outcomes and progress. 
Developmental Psychology, 52 (8), 1247-1261. doi: 10.1037/devO000151 


Phillips, D., Lipsey, M., Dodge, K., Haskins, R., Bassok, D., Burchinal, M., Duncan, G., Dynarski, M., Magnuson, K., & Weiland, C. (2017). Puzzling it out: The 
current state of scientific knowledge on pre-kindergarten effects. The Brookings Institution. Retrieved from https://www.brookings.edu/research/puzzling-it-out- 


the-current-state-of-scientific-knowledge-on-pre-kindergarten-effects/ 


Ramey, C. T., Campbell, F. A., Burchinal, M., Skinner, M. L., Gardner, D. M., & Ramey, S. L. (2000). Persistent effects of early intervention on high-risk 
children and their mothers. Applied Developmental Science, 4, 2-14. 


Reynolds, A. J., Ou, S.R., & Temple, J. A. (2018). A multicomponent, preschool to third grade preventive intervention and educational attainment at 
35 years of age. JAMA Pediatrics, 172(3), 247-256. 


Rojas-Flores, L. (2017). Latino U.S.-citizen children of immigrants: A generation at high risk. Summary of selected Young Scholars Program research. 
Foundation for Child Development. https://www.fcd-us.org/assets/2017/10/F_FCD YSP-Summary 010918.pdf 


Rojas-Flores, L., & Medina Vaughn, J. (2019). Determinants of health and well-being for children of immigrants: Moving from evidence to action. 
Summary of selected Young Scholars Program research. Foundation for Child Development. 


Ryan, S., Grave, M.E., Gadsen, V., & Levine, F. (Eds.) (in press). Advancing knowledge and building capacity for early childhood research. 
American Educational Research Association. 


Schweinhart, L. J., Montie, J., Xiang, Z., Barnett, W. S., Belfield, C. R., & Nores, M. (2005). Lifetime effects: The High/Scope Perry Preschool Study through 
age 40. (Monographs of the High/Scope Educational Research Foundation, 14). High/Scope Press. 


Trent, M., Dooley, D.G., Dougé, J. (2019). The impact of racism on child and adolescent health. Pediatrics, 144(2): e20191765 
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Weiland, C., & Yoshikawa, H. (2013). Impacts of a prekindergarten program on children’s mathematics, language, literacy, executive function, and emotional 


skills. Child Development, 84(6), pp. 2112-2130. doi:10.1111/cdev.12099 


Yoshikawa, H., Weiland, C., Brooks-Gunn, J., Burchinal, M., Espinosa, L., Gormley, W., Ludwig, J.O., Magnuson, K.A., Phillips, D.A., & Zaslow, MJ. (2013). 
Investing in our future: The evidence base for early childhood education. New York: Foundation for Child Development and Ann Arbor, MI: Society for 


Research in Child Development. https://www.fcd-us.org/assets/2013/10/Evidence20Base20on20Preschool20Education20FINAL.pdf 


Webinars 


Getting it Right 2020 Webinar Series www.tcd-us.org/getting-itright-webinars 


Getting it Right: Using Implementation Research to Improve Outcomes in Early Care and Education Virtual Launch (June 30, 2020) 
Getting it Right: Part 1: What more do we need to know about high-quality ECE programs (July 14, 2020) 
Getting it Right: Part 2: Implementation Research in Early Care and Education (August 18, 2020) 


Getting it Right: Part 3: Moving Towards Equity Through Implementation Research (September 2, 2020) 


Websites 


Learn more about the Foundation for Child Development’s Young Scholars Program and view examples of implementation research 


httos://www.fed-us.org/about-us/young-scholars-program 


Learn more about the National Network of Education Research-Practice Partnerships (NNERPP) 
httos://nnerpp.rice.edu 


Learn more about the New York City Early Childhood Research Network Studies 
https://www.earlychildhoodresearchny.org/researchlibrary/projects/networkstudies 


122 GETTING IT RIGHT: THE CONVERSATION GUIDE FOR PREPARING THE NEXT GENERATION OF IMPLEMENTATION RESEARCHERS FOUNDATION FOR CHILD DEVELOPMENT 


FOUNDATION Connecting 


research, policy 


FOR CHILD and practice to 
DEVELOPMENT | “that work. 


www.fcd-us.org 
475 Riverside Drive, Suite 248, New York, NY 10115 © T 212-867-5777 


Suggested Citation 


Foundation for Child Development. (2021). Getting it Right: The Conversation Guide for Preparing the Next Generation of Implementation Researchers. New 
York, NY: Foundation for Child Development. https://www.fcd-us.org/assets/2021/10/GettingitRightGuideforlmplementationResearchers.pdf 


